Eh, I think you can, as long as you are consistent about it. Which is rarely happens, as often definition of "sex" gets casually swapped with "a guess of someone's sex based on their gender presentation" more often than not.

Defining sex with chromosomes is surprisingly useless in most practical arguments.

For example, unless you have someone's genome sequenced before entering bathroom, it's literally impossible to sex segregate bathrooms, under that definition of sex. Hence the whole "bathroom debate" is actually about conformity to presenting "appropriate" gender. But this where swapping the definition of sex mid argument becomes part of many transphobic narratives.