I personally believe a person's intention matters, but many people in this country grow up with deep victim complexes, due to historical and other social reasons. I don't know if you remember when George W. Bush complimented Barack Obama by saying he was very "articulate", black people took that the worst way possible, because "articulate" used to mean another thing when it came to describing black people, who usually didn't speak very properly.

These things are imprinted onto a person's mind from childhood, and it becomes very emotional and part of their world view. The problem is, it may be hard for someone who doesn't know you very well to know if you really did mean it in a racist way. I'm not black, and even I have been exposed to these concepts. I've seen it in movies and on TV, fiction and non-fiction.

In an ideal world, black people wouldn't be so easily triggered by something that may or may not be racist during a friendly basketball game, just as some feminists shouldn't be triggered when you are courteous and hold a door open for them. But, these are the issues of our time, and in my opinion it's all too convoluted to say how people should react. It doesn't make a lot of sense, but it's just part of our culture. The list of things in that category is infinite, and this one happens to pertain to black people.

It feels wrong to apologize or hold back from doing something even though you mean no harm. It's like admitting guilt without being guilty. It's also like granting people the right to control your actions with their feelings, which feels so wrong. But, you may as well choose your battles, and it's okay to sympathize with other people who have different sensitivities. You could try to convince them that it's not always meant in a racist way, and that even you get called "boy" in that way. If that doesn't work, then maybe it's not worth the trouble.