My understanding and the way I see it is that not wanting to have sex with a particular trans person isn't transphobic, but rather making broad statements like "I would never have sex with a trans person is more rooted in transphobia. It's completely excluding all of them without ever knowing them.

I've seen the counter argument made "then gay men not wanting to have sex with a woman is sexist.". And no, a gay man meeting individual women and not wanting to have sex with them isn't sexist. However, gay men going around and saying they'd never have sex with a woman kinda is. It's usually not put that nicely though. Stuff like calling vaginas axe wounds or just bring weirdly extra about women's bodies.