They're not, you're only feeling that because you are white and that's your experience. Or, if you're not white, you're surrounded by a lot of white people and white history, so you know all about the bad things that certain people did to other certain people in your particular region. If you lived in Africa, though, it would be different. There would be an entirely different history and context and the "bad guy" would be someone else. As you sited Mongols, the Mongolians commonly believe - to this day - that the Chinese are evil. I used to be close with a Mongolian girl who carried a picture of Genghis Kahn in her wallet like a picture of Jesus and claimed that the Chinese had separate farms for livestock. One where they fed the livestock healthy food and the other where they fed them excrement and those were the livestock animals they sold the Mongolians.

I mean, so much of this is regional and contextual. It's not about "white people" oppressing or having to feel guilty. It's just about recognizing past historical mistakes, owning them, and agreeing that it would be bad to repeat them. The guilt factor is more or less projection. You see someone bad that happened by the hands of someone who shared your race in the past and it feels unfair to be held accountable for it, the same way people hold Muslims accountable for Muslim extremists or Christians accountable for Christian extremists. We don't want to be held accountable for the shit things our colleagues do, but it is entirely healthy to "atone" for the atrocities committed by our ancestors wherever they may be from. And this whole rhetoric around "privilege" being bad is a fallacy - everyone has privilege, be it race, gender, wealth, location, etc... The point is to recognize whatever implicit privileges you may have so that not only are you aware of them, but you will be more aware when your privilege impedes on someone elses freedoms. It doesn't make you a bad person if you were related to a slave owner and I think most minorities know that, as they themselves probably have some evil ties in their lineage as well. But, it does make someone pretty ignorant when they refuse to even acknowledge it or accept it as a dark part of their races history.

TLDR: That's your perception because of your specific experience and location. In other areas of the world it's a totally different story.