There isn't, but people being uncomfortable about it isn't unfounded either. There's a reason why dom-male and sub-female raises more hackles than the converse.

Linsey Ellis defines and discusses cultural appropriation here. That "it isn't inherently bad or good" but that some contexts cause it to lean away from harmless.
https://youtu.be/2ARX0-AylFI?t=767

I bring this up because it applies to this situation as well. Why is one reacted to more strongly than the other? It's because, historically, women have been relegated to a submissive role and it has taken hundreds of years for Western cultures to begin to stop considering women as 2nd class citizens.

The video brings up an example of "harmless" cultural appropriation where a Bollywood film borrows American motifs for a music video. She argues that because Americans/the West were not historically oppressed by Indians, it doesn't offend the same way that the converse situation would.

Letting something that applies the old, oppressive status quo slide without comment on that quo feels too much like condoning it for some people. Whether or not that is true, I think it's a reasonable sentiment to hold.