I agree completely with your theory and think it's a real problem. Here's an interesting article that touches a bit on this problem. According to the author, 90% of crossdressers are only doing it for the sexual pleasure and only as little as 10% (or less) are actually transsexuals.

The article is a bit long, so here are some key quotes regarding this issue (still long, sorry):

"It took me several years to actually understand what is really going on in the crossdressing world. And it should have been obvious: after all, the ratio of crossdressing friends on my list who are just out there for sexual pleasure, versus those that are interested in exploring a different sexual fantasy, is about 10:1. I just considered that a statistical anomaly. Much later, however, I came across some sociological studies that validated those numbers: 90% of all crossdressers really just dress like women to get sexual pleasure."

.

"So the image that the mainstream audience has is that there is a vast amount of fetishism going on, but it all happens behind doors, in strict privacy, and what people do there is up to them."

.

"Crossdressers, but most especially transgendered people, challenge that assumption. Because the general public sees crossdressing as a sexual fetish — and they will be right in 90% of the cases — they cannot possibly understand why crossdressers wish to do their crossdressing in public. Also, because there are many crossdressers who go as far as to do body modifications (remember the Brazilian case), it’s hard for the public in general to understand why they have to be ‘exposed’ to one’s sexual fetishes in public, when all other sexual fetishes happen behind doors. So even a moderately tolerant person, watching a transexual in transition, might simply ask — why do I have to be exposed to a guy’s fetish of having boobs and showing them off in order to get male partners? Because we’re not subjected to female BDSMers dressing up as dominatrixes on the workplace, why should we be forced to accept ‘transgendered people’ in the workplace, who publicly exhibit their fetish for all to see?"

.

"And, most particularly, when fighting transphobia, how can we make it clear to the public at large that transexuality has nothing to do with sexual preferences, but merely with identity? This is actually a huge problem for LGBT activists, since LGB are all about the right of freely choosing a sexual partner, while transgenderism is about identity. The problem is that just one out of ten crossdressers is really worried about identity issues; all others are just worried about getting sexual partners!"