Two reasons. Fear and existential confusion.
People fear what they don't understand. People's way of being is their standard for the work. These other people operate in a way that doesn't make sense to the person's standard for how things should be. They don't understand. They don't know what to make of this and basic human survival kicks in. If it's an unknown then it's possibly a threat.
Threats are to be feared. Threats are to be eliminated. Therefore these people are threats and must be attacked and removed.
Second, again remember that a person's own sense of self is their standard for the world. People will do what they think makes sense in life. They're going to do what they think is the one, best way to do things. For some people the problem comes when this idea that their way is the best way gets challenged.
These people are living in a way that is contrary to what they view as the one correct way to be. This brings in two possible problems. The first is that the person actually isn't living in the one best way possible and this other, strange person is the one who has it right. The person might be wrong. They might be living their life wrong. But their way of living is their identity. And if their way of living is wrong then them as a person is wrong. People don't want to be wrong.
Second this beings the possibility that there is more than one correct way to live. The person thought that they had the one correct way. There might be several correct ways? How can that work? They were so sure that they had the one correct way. But if both ways are correct then what way should the person follow? How will they know what to do if there's more than one choice? Which way is more correct-er? What should they do now that their one, true path is in question. What else do they possible need to question the correct-ness of in life? Their world gets through into confusion because their foundation gets shaken.
These people unsettle people's sense of purpose and self. Therefore if you remove these people who settle them then the problem goes away. Ignorance is bliss, after all.