I'm not saying everyone is, but a decent amount of people are. It doesn't imply not seeing the other gender as people. I don't know how bi people feel, especially since they are a vast minority and I'm sure some probably do have social anxiety too, but that seems like total digression from the point.

I used to definitely feel much less comfortable around men, and it wasn't even that I was attracted to all men but just the idea that they could think I'm attracted to them or think I'm this or that, it definitely messed with my head more than women. Some people are generally more socially anxious and then the questions that are raised when interacting with the opposite sex can make them cross that line where discomfort becomes big enough to hinder interactions.

I think it's also harder for men to break out of it considering that they are judged more harshly by women when they come off as awkward and that a lot of initiative falls on them. I was able to get out of my anxiety pretty passively since it never caused me bigger problem and eventually I had enough experience with men to learn to curb it. Had men reacted negatively or teased me when I was socially clumsy and shy around them in my younger years, I would probably have developed differently.

Has nothing to do with not seeing a gender as people.