I think the issue with this is that prejudice is separate from sexism. I think that's an issue with a LOT of topics like this. It's all semantics.

Prejudice is ANY preconceived opinion about someone that isn't logical or reasonable. The people assuming men are always violent are prejudiced because they have a preexisting belief that men are violent and put that belief on any man they meet without giving that man an opportunity to show one way or the other.

Any of the "-ism" words (sexism, racism, ableism, etc.) are generally rooted moreso in how the overall social bias negatively impacts people on a much larger scale. Women are less likely to be elected president due to a social bias belief that they are too hormonal to be in a position of power. Women have less say in their reproductive rights because they have less representation in lawmaking. Women are more likely to receive below standard healthcare due to a belief that they are hypochondriacs and some conditions like endometriosis can take up to eleven years to be diagnosed despite a rate of one in ten women having it. In some cultures having your baby born as a girl is so undesirable that women are forced into abortions. These issues are all a result of a much larger scale of bias than what happens on a one-on-one. There are aspects of sexism that do damage men, one big instance being genital mutilation because of a preference for a certain look. However a lot of the issues cited as sexism by men are moreso just small scale social bias. That doesn't mean it doesn't negatively impact them and it still deserves attention. Unfortunately a lot of the discrimination men face is the fault of other men which makes it very difficult to combat because toxic masculinity isn't something easily broken down.

Prejudice often plays a big role in the development of systematic oppression but isn't the sole factor. When people say reverse racism doesn't exist they don't mean that no white person has ever experienced an instance of racial prejudice-they mean that the social bias against that person wasn't negatively impacting them on the same scale as institutionalized racism. These things depend heavily on what groups of people are in control of the larger governing parties and how their bias changes the lives of all the people they have a level of control over.