I don't think a mutual relationship is about tolerating each other's hobbies. You shouldn't have to sit with your girlfriend/wife and watch the drudgery of television because you are or feel obligated to. I think it is more about exploring joint hobbies together, things that interest and appeal to the both of you. Self-interest isn't that inviting and puts a ceiling (or floor?) on the depth of a relationship.
It's hard though. It takes a concerted effort to involve and open yourself in the other person's life that much. And still, some down time is essential, I definitely get that. It's just that every time one of you chooses to pursue a time consuming hobby that the other person isn't interested in you're basically excluding the other from you. I think after so much of that it begins to tax a relationship. And just because you might set ground rules at the beginning doesn't help.