Treating the two continents as monolithic cultures, and then having one enslave the other? Those are deliberate choices, and people will interpret them as deliberate.

Race-based chattel slavery is a fairly modern capitalist invention, first created to support sugar plantations. The racism was born from the industry. 'It's okay to do this because they're not really human', etc. Certainly racial bigotry pre-dates the trans-atlantic slave trade, but it didn't go beyond basic nationalist and tribalist ideas until it was profitable.

All this to say, what is the basis of slavery in your world? What is the historical context? Cui Bono? Slavery is a very serious topic and needs to be treated as such. In a historical context, slavery as punishment and as a profit of war is very common, but not at all delineated by skin color or continent. So you have to ask yourself, why is it so specifically delineated in your world? Why don't they take slaves from their own people, as well? Do they believe that the other people are lesser beings?

I don't think there's anything wrong with the concept, but I do think you that, when dealing with such charged subject matter, you need to be aware of why you're making the choices you're making.

Good luck.