But it's worth considering where this perception comes from.
I think you're ignoring that there has long been a concerted effort on the part of people who oppose women's rights--whether that be opposing suffrage or opposing feminism--to hand the fringe or elements the mic and paint them as representative of all feminism. There have long been dedicated attempts to paint women's rights activists with all sorts of negative adjectives and to advertise the fringe as the heart of the movement.
We're talking about a long-standing academic discipline and political movement, and yet the first examples people usually reach for are youtube and tumblr personalities. I'm not personally the authority on who's really a scientist or a literature professor or a theologian, but I still don't turn to social media to find an authority on physics or English literature or comparative theology, and I wouldn't consider whatever I find there to be representative of those fields. And yet we have internet, news, and radio personalities like Rush Limbaugh whose aim it is to paint these largely uncredentialed, frequently anonymous, usually "armchair" elements as mainstream, and then use that as an excuse to either distract from or oppose mainstream objectives or to justify harassment.
Furthermore, it's not a coincidence that the negative stereotypes people have of feminists--the ones they often ascribe to the "trolls" and then extrapolate to the "non-trolls"--are the same negative adjectives people 1) preferentially use on women who speak their minds and not similar men, and 2) have been perpetuated about women's rights activists since the time of the suffragettes. It's awfully on the nose that these smears are the same ones that have been slung at women asking for even the most basic rights--like the right to vote--for generations.
The reason I think this is important to note is that if this perception is due in part to a concerted effort by those who oppose feminism, abandoning the word or symbol probably won't help--you'll just get "euphemism creep." The issue is there have long been people who oppose the objectives of women's rights activists and use these kinds of associations to avoid having to actually engage.