Going to say something that will make people cringe but:
In the opinion of a lot of people, keeping human slaves was morally acceptable because morality is subjective. They saw those people as being sub-humans with less rights than them. They wouldn't morally keep a white man as a slave, because their moral compass was different. A lot of them probably thought they were good people.
I don't agree that morality is objective. I think some people falsely think that it is objective, but I do believe there is a "right" and "wrong". Killing people is wrong, even if some people think it's ok because they think morality is objective, it's still wrong.