I don't think you need overt acts of racism to feel avoided/singled-out. Like being told 'oh, this is how you wash your hands at a jinja - it's rude if you don't do it right' (even though you'd been to 1000s of jinjas, know the damn routine, and see your Japanese buds half assing the routine every time).

That's not even racism.

You also hear a lot of 'gaijin' murmurs from drunks in Shinjuku etc late at night.

Experienced this, but then again - drunks in Shinjuku. And not every mention of "gaijin" has to be negatively connoted. Going into an izakaya alone as a foreigner will naturally invite weird looks. I've met quite a bunch of nice drunks in izakaya's who were just interested in me because I am not Japanese. Their "gaijin murmurs" turned out to be out of interest, not malice.

A toddler called me out on my fat schnoz at an aquarium once but his mum scolded him and it was cute so I'll let that slide.

As opposed to what, socking them in the face?