I’ll admit I misheard something he was saying one time but like that doesn’t take away from his abilities as a doctor.

I think this is crux of the issue. People don't want a "white doctor", they want a doctor they and clearly understand, and they for some reason think asking for a white doctor is an easy way to do that.

While mishearing something doesn't take away from the doctor's ability, it could take away from your ability to carry out what they are asking you do. The end result is still the same, you didn't end up with the same level of care, because you weren't able to communicate effectively.

I've had good and bad doctors of all races, while, black, Indian, asian... some were bad because they just didn't seem to care (the last white male doctor I had), and others I just assumed were bad, because I couldn't really understand them. They may have known their stuff inside and out, but if they can't relay that knowledge to me, it doesn't do much good. Communication is a very important part of being a doctor.