But isn't there something fundamentally different about a group of people loosely associated around a particular thing they happen to have in common - not believing in god or liking a particular kind of sex or, for example, being a fan of a particular sports team or piece of media or whatever - than groups that lie on axes of oppression?
The difference comes into play with how hostile society as a whole is to that group. If Football fans started getting killed for their enjoyment of the sport, were actively silenced by non-fans, were used as stepping stones to gain power in society, etc. we would call that an axis of oppression. Yeah it seems ridiculous, but given the other bases for oppression, none are any less arbitrary. If you go with the route that no oppressed group chooses to be a part of that group, that seems like it's getting into biotruthy marginalization of people being themselves regardless of how optional that is.
#NotAllMen is a problem because it's actually slightly right, but is used to sidetrack discussions about bigger issues. This is where the fallacy of relative privation ceases to be a fallacy, since smaller issues are used to detract from larger ones. Unsupported stereotyping of any group is bad (even unoppressed ones), unless it's implied by the definition of the group, and #NotAllMen calls attention to that and the fact that toxic masculinity and patriarchy is blamed on men in these discussions, which is also partly wrong. But it is a problem because it has a thread of truth that is used to divert attention away from larger problems.
To phrase the question another way, how much can a community be defined by it's most negative aspects or worst members?
When either the defining factor is itself a problem or implies problematic things, OR when a significant portion of the community expresses problematic traits (in other words, there is a correlation shown between belonging to the community and having X terrible opinion). We can stereotype redditors as terrible people in Prime not because membership requires being shitty, nor because most users actually terrible people (they aren't), but because the majority allows terrible opinions to flourish without contest and sometimes encourages them while discouraging contesting them.
So I guess to answer your title question, the only time it's okay to generalize is if the trait that's being generalized is actually shown to belong to most of the group or is implied by membership to the group. And, of course, the generalization should be relevant to the topic at hand.