Whether men and women are biologically different is an empirical matter, so I won't argue for against it on principle. My intuition is that it's true, but again, it's something experiments will tell us so I won't try to argue the science of it here. That said, I'm a bit lost in your reasoning. You're saying that there are plenty of exceptional men who have womanly traits and women with manly traits. Certainly. But that does not disprove the possibility that there are general differences between men and women. Let's say women are 10% more caring and empathetic on average. There will still be plenty of exceptional cases of cold, mean women, but a group of women of any decent size is bound to be more caring than a similar group of men. Your single anecdote about your own personality doesn't really matter. It's a sample size of 1, and from an unreliable source (somebody's own idea of themselves). I'm a scrawny, sensitive guy who is much less masculine than most men. Of course people like us exist, but sexual dimorphism is about trends.

Nobody ever has any issue talking about sexual dimorphism in lesser animals, but the second we start talking about humans, politics creeps in. Start with some very clear examples: strength. This is one nobody can deny because it's too stark, and too far removed from the effects of socialization. Nobody would ever say men and women are equally strong, even though a female body builder could destroy a whimpy dude like me in a fight. Same thing with genitalia. And height. And facial construction. But the second we start talking about a more abstract, mental characteristic, it is immediately deemed impossible that it is inherent, because of the perceived political implications. Now, it's also not correct to say that just because strength is an inherently unequal between the sexes, so is propensity to nurture. That does not follow, and I'm not making that point. All I want is for that possibility to be discussed, rather than just shut down as 'sexist'.

The boundaries between male and female are man-made, rendering every social difference we make between them absolutely pointless and devoid of biological sense.

The boundaries between men and women are decently clear. Not always, as biology is rarely black and white, but enough that the vast vast majority of humans categorize men and women the same way. Certainly more clear than 'hot' and 'cold' (just listen to my girlfriend and I argue about the temperature :p), and those are still useful categories. Fuzziness also does not render differences 'absolutely pointless'. Fuzziness is inherent in the way the human brain constructs categories. A category as 'a set of things that share one set of characteristics' is a very naive, dated conception of categories, yet I can't help but feel like it's the one used in these discussions. We can't draw a precise line between 'soft' and 'spiky', but if you asked 100 people to classify a set of drawings as either, you would not get a random 'pointless' distribution. Most human categories are this way, and them not being black and white (something nobody has argued for in the last 200 years) does not render them 100% arbitrary. There is a lot of space between 0 and 100.

Moreover, if indeed the difference in what is perceived as male and female traits was linked to hormones, these differences would be universal and every society would have the same set and would have had the same since the dawn of time, and it is just not the case, at all.

I disagree. It's very clear that a) biology shapes society and b) there is still a lot of room for tradition and custom to work on top of this. Sexual customs are not 100% the same throughout all of history, but we're clearly biologically programmed to seek out sex.