I agree with you on this point. There are plenty of characters that are created entirely to be flat, non-fleshed out comic relief or side characters. Fanservice in the case of these characters can certainly detract from the show, but the character is already unimportant or unrealistic, so there's really nothing more to hurt. When you have a main character being sexualized, like Karen in Code Geass, to use /u/7TeenWriters's example, it hurts the character and the show considerably more than otherwise. It might cause you to praise the show for creating a character so real or well-written (not necessarily the same thing) that his or her objectification or out-of-place sexualization actually hits so hard, but I would argue that that's precisely why it matters so much. In a show with a bunch of poorly written, characters, out of place fanservice may lower the show even more, but the show is still bad. You're not watching it for its great characters in the first place, and it's probably not something you're wont to recommend anyway. In a good show however inappropriate use of fanservice detracts from what might in every other aspect be a good show.

I'll use some examples of my own. In the veritable ecchi masterpiece Onegai Twins (yes I'm being sarcastic), which just so happened to be my first non-childhood anime experience, there are plenty of poorly written characters. There are some whose sexualization is in line with their character and expected, and like 7Teen said, it doesn't really add or subtract. There are other cases (though perhaps the ecchi tag makes it arguable) where the fanservice of a character goes against the even small amount of characterization they had. This brings down the show even more, but it doesn't matter. No one is arguing that it would have been a masterpiece otherwise, or that the show is portraying women badly now. It already was. On the flip side consider one of my favorite shows, Revolutionary Girl Utena. The show has themes of adolescence and romance, and has several characters who have sexual experiences or are seen in a sexual light, although I would argue tat there is nothing in the show that could really be considered fanservice. The main character, however, is never sexualized. It would be completely out of character to act or even dress suggestively. So imagine that all of a sudden the show takes pauses after one of Utena's arena duels, to show a gratuitous wardrobe malfunction in which she fumbles with her sword and cuts a revealing chunk out of her uniform. She's blushes and the scene ends in a drawn out "kyaaaaaa!"

Putting aside how hugely improbable that would be considering the rest of the show, would this scene ruin the show? No, of course not. Would it keep it from being a masterpiece or stop me from recommending it? No, not really. Would it be funny? Well depending on what you find funny, possibly. And it certainly wouldn't lower the show to the level I'd at which I'd place Please Twins. But the damage to Utena's character, to the show's credibility, and perhaps especially to the narrative focus on a strong female character would be much larger in a good show that it would be in a show that's not likely to leave any positive marks on the viewer.