The difference is, if you get called a racist or a slave owner, your feelings get hurt. That's all that happens. You can still go to stores without being followed around by security. You can still get pulled over for speeding and not worry about getting shot. You won't get turned down for a job interview because your name "sounds black" or you have "ethnic (code word unprofessional) hair". You likely don't have grandparents who remember not being able to go to certain stores or be able to eat certain places. You are probably more likely to know how to swim, because your parents learned from their parents how to swim at public pools. Your black friends grandparents weren't allowed in public pools.

I'm white, I have been called racist many times, and I'm not going to lie, it hurts my feelings. But I can deal with my hurt feelings. Saying that a person calling you racist is just as bad as what "dark skinned people" have gone through, and continue to go through, is ridiculous. But this is Reddit, so I'm sure you'll get all the support and validation you need