I think that people do care what men look like, but that society tells us not to care. If you do more, it's acknowledged that you're doing more, but if you do less it's not unacceptable.

I think it kind of comes from there not really being an "ideal" man in media like there is an "ideal" woman. That's not to say there aren't certain traits that are more common or that men don't feel pressure to look a certain way anyway, but there is no required body type to be considered attractive. A buff man with a six pack can be a heartthrob, a 50 year old man can be a heartthrob, a lanky skin and bones nerd can be a heartthrob. And there are numerous examples of each. There are also a lot of unattractive men with insane charisma that are also considered heartthrobs. So I think that adds to society not really "caring" about whether a man is aesthetically pleasing because we're exposed to so many different male role models that are all "acceptable" to be attracted to. So just being presentable is enough, because we've already been spoon-fed by the media into thinking of a wide variety of men are attractive for a number of different reasons.