All right, sonny, it seems like no one ever taught you something, so I'll give you the lesson. Culture isn't a wrong or right answer most of the time [we tend to think it is when a culture demands really bad things like extreme inequity]. You already know that, but you're taking that fact to a place it doesn't have to go.

Even though there is no right or wrong answer, the simple fact is you are born into a particular culture with a bunch of people around you holding large veins of similar culture. When someone in that group ventures away from the original culture by a wide margin -- when their behavior is distinct from a hobby or taste within the culture but instead is a different culture itself -- you fundamentally insult everyone holding the original culture. It's as if you say, "Yeah, I started with this culture, but I prefer this other stuff waaay more. Thanks for the offer." And since culture is a personal thing, something people feel personally and strongly about, it will ultimately be insulting and cause a negative response.

It'd be like your brother saying, "Yeah, I know I was born into this family, and frankly, you guys are not bad or anything. But, I'd rather travel over to my best buds family instead. I just prefer them!" It feels like betrayal.

The guy who is born into American culture and then completely swaps into some envisioned subset of Japanese culture does that very thing, so it's natural for Americans to feel something is off.

But the more important point is, from what I hear, even Japanese people hate anime-obsessed white people trying to consume everything Japanese. It's synthetic. Just stay with your own family, you'll never have blood ties to your best bud's household.