I know that it can be easy to slip into that way of thinking. But if you see the way groups like incels talk about women, I see the same kind of bitterness that's aimed at an idea of what a woman is based on statistics, stereotypes, and specific anecdotes, but that's not an accurate picture that represents all of the massive number of individual women that live on this planet. Men and women are two huge groups. If it's unacceptable/illogical to judge based on race, it should be even moreso to judge based on sex.

I think we're fine with these public spaces as they are. Places like prisons have always been strictly sex-segregated. Trans people have been using bathrooms and changing rooms at their own discretion and nothing significantly horrible has happened to my knowledge.