I'm a physician and I work with a lot of different kinds of people from all sorts of backgrounds. Some people that have a lot of money and dress nice, and are well educated and polite. Others who are less well groomed, less educated, or less mentally stable.

This question is really hard, but in my view there's a difference between treating people with respect and dignity no matter their background/appearance vs trusting them, spending social time with them, or just generally giving them sensitive or important information or access to your personal life.

Humans are a very social species. If a person is not able to clearly signal that they are trustworthy based on the obvious societally accepted standards, then it is very reasonable to be cautious around them. Is it sad that we get this feeling around certain types of people? Yes. But just because we can recognize and empathize with certain people or groups that doesn't mean we can't protect ourselves.