The thing that annoys me the most is that the whole toilet debate is just a motte and bailey argument. It's always "I've transitioned and all of my primary sexual characteristics are identical to those of a woman, why shouldn't I be allowed in the women's bathroom?"
I don't think many people have a problem with a fully transitioned person using the bathroom that best fits their visible sexual characteristics. People have a problem with a gruff middle-aged man being allowed into a female space because he has declared himself to be a woman. I don't think people have a problem with it because of fears of sexual assault either - it's literally just about women having a place to adjust their bras and touch up their makeup in (relative) comfort.