I agree. As an AA dude I've felt had that problematic perspective about white women myself, and I think it has to do with a few things. Part of it was the feeling of subverting racist expectations about Asian men, and who society assumes they are "capable" of dating. And, I believe it has a lot to do with the other camp of Asian dudes who complain about Asian girls who only date white guys... people who put off that resentful "off" vibe you're describing. I think the problem at the core of BOTH of these positions, which at their lowest will overlap, is fundamentally an insecurity about Asian male identity and perhaps an anxiety about it being attacked - of which, admittedly, there is a history. My takeaway was that regardless of how one reacts to the presence of racism in dating, "subverting" it or otherwise, it means our mental and social activity is still circulating around a set of cultural assumptions that shouldn't be a factor in the first place. It's never good for that subconscious insecurity to affect the way we pursue intimacy. But first I think it takes honesty and developing a strong sense of self, which for me involved a lot of inquiry about what it meant to be Asian-American.
Also, I think my dating-white-girls situation was not too different from what Asian women experience themselves when they express, either verbally or socially, a "preference" for white guys. For me at least, I think it's the fact that I was being read differently as a person based on my proximity to whiteness (as Melissa Harris-Perry once put it on NBC). Instead of being racialized with an "Asian _____" prefix, walking around with a white girl meant I would be interpreted differently - with surprise, and, I hoped, with more consideration for who I was as an individual or "normal" person. Sometimes white people would even smile because omg society is changing! Because of me, wow! It's so silly but that small kind of stuff was a factor. Like sometimes even now, I will be walking through town with my close friend (who is a very white looking girl) and there's the wide range of small reactions that are hard to ignore, simply because they assume we are dating and mentally they are shipping us hardcore. I think these social cues, or even an imagined anticipation of these attentions, affects people a lot more deeply than we'd like to think. And that's why "preferences" are problematic - because they're not just natural or passive settings: they are very, very socially produced.
I could probably rant about this stuff forever but my conclusion is just... you know when there's bad energy. Like, if it feels like there is a negative vibe, even just a feeling, then there's probably something that needs to be scrutinized internally.