There's more than two possibilities, sometimes how word are used in a society can convey a meaning via subtext that isn't stayed outright.

This is what people who complain about toxic masculinity are getting at. I've seen multiple definitions of toxic masculinity, but whatever they are they are always things that could be generalized across other societal groups. Yet in societies' dialogue it is rare to see the label of "toxic" applied to anywhere but men. Even in this thread you can see people discussing whether toxic feminity exists. So yeah, it's largely sole use towards men does end up feeling like a bit of a jab towards men.

Or to make it more plain, all cultures have negative aspects to them, yet suppose we lived in a society where when discussing the negative aspects of African american culture, people routinely used the phrase "toxic blackness", but rarely described white, or asian or India etc culture as such, wouldn't this make you uncomfortable?