I honestly don't think any man thinks that way when choosing or not choosing a novel. I think it's just a matter of relatability. When guys think of a "girl book" they think of romance and drama, which some guys rigidly distance themselves from, and also vice versa, some girls prefer that sort of thing and would rather stay away from what would be seen as a generic action/adventure book. I've never ignored a book because of it's target audience, but sometimes the content simply doesn't intrigue me.

I'm also fond of seeing people broaden their domain in terms of genre and author, so those forementioned posts really make me smile. :) Feel free to disagree, however.