There's several reasons simpler and more practical than a grand fantasy. I think it's possible to prefer trans women without invoking any of the below, but the reality is that the vast majority of mtf-attracted people are guilty of at least one of these, hence the negative connotations on the chaser label

1) Trans women are generally deeply troubled by their masculinity, and don't want to be attractive to people who prefer the existence of that masculinity. They generally want someone who finds them attractive despite it, not because of it.

2) Chasers are generally looking for sex only. Some women like anonymous sex, but chasers often have baggage. They might find their own desire shameful and only treat her as a "dirty secret", or have very little respect for her in general.

3) Chasers (stereotypically) seek being penetrated and giving oral sex, things that highlight the source of her affliction. Trans women often, as children, would pray for God to turn them into a cis girl as they went to bed. Do you think such a person wants people slobbering to get at her deformity?

Of course, there's many types of trans people, and some that don't hate their genitals and accept being non-binary. And some that aren't put off by chasers. But these reasons are why the typical trans woman generally avoids chasers.