Secondary sex characteristics aren't strictly binary, and clearly part of gender stereotypes.

What is the point of this? Almost nothing in nature is "strictly binary", and yet we are still capable of talking about natural categories that obviously exist and matter. It is impossible to draw a perfectly sharp and explicit line between the land and the ocean, but that doesn't mean the land and ocean aren't real and meaningful categories, or that we can't classify 99% of the earth's surface into land or ocean.

Facial hair, height, boobs, voice are all characteristics people use to identify gender, but they aren't clear markers of biological sex.

Just because you can't use a trait to assign membership in a category in every individual case doesn't mean they have no clear correlation with one category or the other. The overwhelming majority of all people with beasts that can lactate without artificial medical intervention are female, and the overwhelming majority of all people who can grow a beard are males. Pretending these patterns don't exist is silly.