I’m just not romantically attracted to other men.

Have you met every single man? If not, that's the point people raise an eyebrow.

Just writing this out because several of my male friends (2 white and 1 Hispanic) have pointed this out and said that my preference was “racist,” which I don’t understand.

It's one thing to notice a pattern of your past, it's another to create a rule or make a generalized statement about it. Like when you say "I’ve never thought of dating/marrying a non-Asian man" we'd be inclined to ask "Why not?" What is it specifically about asian men that makes it inconceivable to date outside those rigid borders?

If it's just their looks, that's fine, but then be consistent with your claims: you don't find men of any other race attractive whatsoever. That, in and of itself, sounds a bit odd—at least highly unusual—but I'm not you, so I can't possibly accuse you of lying. It might actually be true, and there you go. Doesn't really matter what other people accuse you of if you know within yourself what you are attracted to or not.