I'm arguing that the rarity of intersex people makes them a notable exception to a general biological principle. Chromosomes determine biological sex, except in a select few cases (which often involve abnormalities with the chromosomes themselves). Homosexuality – sexuality in general – does not have an impact on the biological sex of an individual.

The point being that you could about as reliably predict the sexuality of an individual by looking at their chromosomes as you can predict their sex, and yet such a proposition is clearly recognizable as absurd. You could look at someone's chromosomes and say "this person is probably a straight woman" or "this person is probably a straight man", and you'll only be wrong on one or more counts about 1/25th of the time, yet it's still utterly useless as a defining measure of sexuality or gender because the exceptions account for millions upon millions of people.

We need to separate biological sex from expectations about personality and behavior

Neurological gender doesn't dictate anything about personality and behavior, beyond apparently including structures that help filter gendered socialization and development of self image: a trans woman internalizes female socialization for her culture and a trans man internalizes male socialization for his culture. The only difference there between the experiences of a cisgender individual and a transgender individual is the active socialization received, where the socialization of the opposite gender is beaten into them through peer pressure, admonishments, and often actual violence, though rarely takes beyond just a weak, miserable mimicry of it that's eventually shed as the individual accepts who they are.

Is there any evidence to suggest sexual differentiation in the brain that is mismatched with genitalia? Hormones are a result of chromosomes, which also form genitalia. You're now arguing that there's a state at which the body develops genitalia and the brain is bathed in corresponding chromosomes, and yet somehow still some part of the brain feels that the body's biological sex is "incorrect."

I linked a metastudy that looked specifically at observable edge cases and their effect on the eventual gender identity of the individuals involved, finding that reassignment at birth did not alter the individual's gender (meaning they were XY individuals assigned female due to damage sustained from cloacal extrophy, who later asserted that they were male and transitioned to live as men), but in cases where an individual had a congenital condition that affects hormone production (in this case congenital adrenal hyperplasia, wherein the adrenal gland of an XX individual produces excessive amounts of testerone) or had prenatal exposure to a drug with estrogenic properties (diethylstilbestrol, an anti-miscarriage drug), there was a marked increase in the incidence of a gender identity other than their AAB sex (5% total for individuals with congenital adrenal hyperplasia, and an extreme 30% incidence for XY individuals exposed to disthylstilbestrol, as opposed to the observed baseline of .6% for the general population).

The results of exposure to diethylstilbestrol are particularly interesting, because they functionally form an accidental recreation of research that's been performed on lab animals, only in humans, demonstrating that the same principles observed in in-depth observation of lab animals are at work in humans as well.

And just to clarify: chromosomes don't control hormones, they (or more specifically just the SRY gene) control the differentiation of the organ that then controls hormone production, and while the hormone triggers that can cause neural androgenization are pretty reliable, they still fail (either occurring when they shouldn't, or not occurring when they should) for an unclear reason in about .6% of cases. It could be the result of an as-of-yet unobserved medical event, or it could just be the probability innate in any biochemical interaction, where the trigger fires but just doesn't reach enough receptors in time, or the low amount of testosterone present during female development just happens to trip enough receptors to trigger androgenization, and then, once the window's closed, the individual's neurological gender is set in stone, whether it matches their chromosomes and body or not.

The idea that a female body cannot be male is not a "collective image." It is a biological fact.