In my experience, the more a hive has to keep them busy, the less they worry about. So they tend to get aggressive when they have no food, water or something else. And, of course, you can make them aggressive. But they don't have a 1-1 awareness of things. So they don't realize you are the bringer of smoke, just that there is smoke. I fog my bees with FGMO and they have no idea who is doing it. They just don't like it. I fill their entrance feeders and they have no idea who this kind and benevolent is, either.

The 'worn shirt' idea wouldn't seem to work, IMHO. They have a short life span and only spend a couple weeks outside the hive before dying. They aren't pets, but rather insects. And they have a short memory. They won't get accustomed to your smell and ignore you, since they ignore you most of the time anyway. It's what you do that they pay attention to. If they feel attacked, they will defend. Otherwise, they just don't really care.

In my experience, scent attracts them more than anything. If I chew spearmint gum while in my veil, they flock to the veil. I tend not to chew gum outside of a veil, though. Women with fruity shampoo/conditioners tend to get them in their hair.

Bee's will 'ping' you to get you away from the hive. This is a pre-cursor to being stung. So if they start ramming you, that's your warning to move away. Of course, sometimes they just run into you, but if you notice you are getting hit a lot, it's time to move away or give them something else to worry about (like smoke).

All that said, please send photos of your 'worn shirt' scarecrow next to your hive. If you can make it to the scale of a bee, even better. Or maybe a veil hanging on a post with a placard that reads 'Never Forget'.