I mean, the concept that the left hand is controlled by the right brain hemisphere and the right hand is controlled by the left brain hemisphere is generally accepted, no? And that the different halves of brains are responsible for different areas. For example left handed people are more artsy, and right handed ones are more logical. I learnt about this from my mother when I was a kid, since my younger brother is left handed and I'm right handed. Nothing bad came of it.

Tying a personality trait to a physical quirk that is NOT TRUE is bad and if doing so is fuelled by hatred, that's extra bad. But this does not mean that certain things couldn't actually be tied to certain physical traits.

But regardless, gender is ultimately not even a physical thing, that's kind of its point, that its more than just "penis people vs vagina people" (that's sex). And yet, I think its safe to say that people with penises are more likely to be men and people with vaginas are more likely to be women - there is a correlation there. This in and of itself is tying things to physical traits.