By that analogy, wouldn't it be fine for white people to use the N word since all people (including white people) descended from common ancestors in Africa? So all the common ancestors are also allowed to use the term regardless of social factors? Or in the other direction, does that mean black people can use racist slurs for people in Asia, the Middle East, etc since they share a common ancestor?

Regardless which direction you run the ancestry, it seems you're now saying that it's acceptable for some out-group people to use racial or xenophobic slurs. Surely even you don't think that logic makes any sense. Even you couldn't believe something so transparently racist. Even you...