Frankly, I think the anxiety revolves around the stunningly common sentiment "believe all women". As others have said, I (a man) feel physically able to defend myself and emotionally and financially independent enough to leave an abusive relationship. However, many people believe any claim they hear and spread it with malice. I've never been accused of sexual misconduct, but I once had a friend's sister threaten to "accuse me of assault" if I didn't kiss her. I obviously didn't kiss her, booked it out of there, avoided the friend and told my parents.

Allegations of racism, sexism, assault, animal abuse, etc. can instantly and permanently ruin someone's life with only circumstantial evidence (one person claiming it's true) or no evidence at all. No decent person feels anything but revulsion for abusers, but men and women both are capable of lying, and though legal charges may never be brought, stigma can be forever. Therefore, making false and misleading accusations of those crimes is also disgusting, as it potentially ruins innocent people and weakens the claims of true survivors of abuse.