yep, they did. And every time they did there was some "world building" behind it. They did durring the roman times because northern Africa was part of the Empire, so black people weren't a strange sight/concept. Also, they viewed things differently back then,citizenship being more important.
They also did during the American Civil War (again, lets ignore the many times they didn't ...to make your example work) because Lincoln was forced to promise freedom to the slaves just to win the war. ( also, let's ignore that there where very clear black/white companies most of the time).
As for WW1 and WW2, the Africans where mostly exploited by the french and others as cannon fodder. Again, a very complicated issue.
I stand by my point, your idealism makes it hard for you to accept these things. You think because you don't care about skin color or sex nobody does. You also have the benefit of "worldbuilding". You live in this century and you are from a first world country(probably). You have a history of atrocities and injustices committed against those deemed lesser. You also have the intelligence and education to be a better person.
However, this is not the norm. In a world where you live all your life in a village with 100 other people that look like you. In any world, european or imaginary, it's impossible to be like you in those circumstances. And if it happens , it is an exception. This doesn't apply only to black/white people, this applies to us vs them, very basic tribal stuff, ingrained into our biology.
IF you want to add an "other" to your world and make everyone accept him, you either change human biology or justify it with world building. That's why you can't have a gay black man leading an army of British without anyone ever caring...
well, you can, but then it's just bad writing.