Like how feminine are we talking? Is it any guy with a slightly feminine tone to his voice or a guy that really leans into the femininity?

If it's the former, I'd agree with the ideas that part of it is what society tells you men should and shouldn't be like, and that there could be internal discomfort with the unconscious idea that being around this will rub off on what others think of you maybe.

For the later, it'd be those, plus maybe just a different level in how much chill a person should have. I'd compare it to other people you know that are a bit of characters in different ways. Like if we were talking about people being a bit country, would you feel fine with someone being a bit rednecky in terms of huntin & fishin, but then be uncomfortable with a guy that leaned into it to being like Larry the Cable Guy wherever he went? In that situation, it could be partly about how an introvert handles people that draw attention and it could just be that you are uncomfortable with lots of behavior that brings a lot of attention.