Sex is socially constructed, again going back to the reason of why we label things - to understand them as a whole.
How is that though? I thought sex was reflective of genitalia? Wouldn't gender be the socially constructed thing, since it has to do with "how you feel"? My only point is that the notion of "gender" (or whatever the word is for feeling like a girl/having your brain wired like a girl) can't philosophically mean very much, if anything. I feel like we're talking past each other a little bit also; I will try to explain my exact point of confusion precisely:
"Boy" and "girl" are only meaningful with the genital referent. If the most meaningful descriptor had to do with brain structure, then do you really suppose that there would only be "boy" and "girl" and not a million other things? In truth, there would be a million other things, and the ideas of "masculinity" and "femininity" would be meaningless. But we know that "masculinity" is a thing; it is the way that people with penises from some particular culture tend to behave.
In these ways the most meaningful descriptor of a person when it comes to being a boy/girl can't come down to brain structure. It doesn't matter that there are differences between boy and girl brains.