My brother has expressed micro-aggression type racist stuff before but I wouldn’t jump to full on racist when I think about it.

Expressed racist opinions = is a racist for all practical intents and purposes.

In any case, I genuinely don't see that there's any room for interpretation of your brother's comment. I'm not sure there's anything to be gained from starting an argument about it at this stage, though, it sounds like brother is pretty set in his ways (this is not a comment someone could make 'accidentally' or in ignorance, he had to have really thought about his 'joke). Maybe just add this to your knowledge about your brother, be ready to challenge him in the moment next time and/or consider distancing yourselves from him (esp if there are kids involved).