OK, weird guy in a fedora. Or 'trilby'. Or whatever. Its not the hat that makes you seem strange to people, its that for some reason you appear unable to pickup on social norms. If you are wearing a fedora, the appropriate attitude for social-existence is: "I'm wearing a unique hat for this era, because it makes me feel a certain way that I enjoy." Instead, people like you try to forcefully redefine social norms to suit your inner-identity of "classy dude", and it upsets you that the world isn't receptive.
Its not so strange behavior to put an out-of-the-ordinary hat on your head. I feel you. I like to wear a silly snow-hat sometimes. I do it because it makes me feel like me, but I put my hat on my body with the full knowledge that seeing a hat that looks like mine is slightly out of the ordinary, and that people will comment/form opinions on it accordingly. I gladly accept the "hey whatsup eskimo", or "isn't it too hot for that hat?" comment sometimes, and feel happy that I have been given the opportunity to interact with someone new and smile and say "yeah I love this hat! isn't it rad?!". Full stop.
What is strange behavior, would be me spending any amount of time considering other people's "misconceptions" about what I'm wearing, let alone voicing those thoughts. All of a sudden I'm a person defending my fashion choices, quite obviously out of insecurity, and my behavior in general is on a slippery slide towards completely antisocial. "Can we stop calling these hats eskimo hats? Eskimos don't even wear hats like this blah blah blah blah." Or thinking about how other people should stop "hating" on the things I wear. That is just a completely sad mode of existence, and fedora wearers seem to have it mastered for some reason. I mean literally, I find it sad that you feel its necessary to defend your hat to strangers. You even go as far as saying "you people". As if you are separate, or even superior in someway, to people who don't like your style of hat.
You should just get it through your skull, brimmed hats are not seen as "classy" anymore in our society. Ask any man or woman on the street, fedoras are officially out-of-the-ordinary and slightly weird. They are the icon today of hip hop singers, interesting movie characters, and of strange people who have difficulty socializing. This is not a problem with fedoras, or society, or anything else though! Its just a fact. Just wear your fedora, stop romanticizing the freaking 1920s or indiana jones or cowboys or whatever, and realize you can let it be. You'll instantly be way less of a weirdo. I understand you put the fedora on thinking "I want to have a little class", but that simply isn't the case, and your rambling hat-defense just reinforces the idea that you are indeed a weirdo in a fedora, rather than a cool-motherfucker like Indiana Jones. The only time you can pull off a strange hat is when you have a swagger and confidence that doesn't depend on external reactions to your hat. Hiphop artists lately who have been making wide brimmed hats trendy, can do it precisely because they aren't fucking weirdos who went "blah blah indiana jones blah trilby trilby fedora I HAVE CLASS DONT U SEE?" every time someone commented.
It is incredibly simple for normal people to pickup on social norms, and then choose to defy them consciously or choose to participate; but your post expresses that you lack the ability to do either. And thusly, I'm sure that many people find you quite off-putting and strange. But of course, thats there fault because they "just dont get it" lol