I'm brown-skinned so I'll take a stab at it. But I'll depart a bit from the other comments and try to abstract outward a bit. Bear in mind, I'm going to write in a simplified way since this is ELI5, but I'll use some mathematical notation too for those inclined to think that way. If you're not, feel free to skip over that.

To me, calling someone "yellow", "brown", "black", and yes even (again, to me) "white" feels somewhat dehumanizing. Let me explain. Think about "Identity" as being a latent trait. "latent" means you can't actually observe it--in fact, it might not even exist as an objective thing, it's just some underlying, unobserved concept that you're trying to capture.

So what are we trying to capture when we call a person, say, "white"? We're trying to capture their identity: saying "he is a white person" is a very shorthand way to describe a person. Now there are potentially hundreds of ways you can describe a person, but fundamentally each thing is one of many traits that can ultimately be used to describe someone.

If you're a math geek, you might say that x=person is a latent parameter and y=traits is a vector of noisy, but observable, features that ultimately translates the latent parameter x into some kind of observable feature space y by some loading matrix/function z:

y = z(x)

If I'm describing you to my friend, I'm trying to give them a sense of who you are as a person. So I might say you're "smart", or I might say you're "quirky", or I might say you're "well-meaning, but a bit oblivious". These are traits that can be immediately personified: all of these things are easy to translate into human form, and it really does the job of "humanizing" you as some latent, ultimately unobservable "person" that you actually are.

So what purpose does calling you "black" serve? Think about it. "Black" is a color. It's not a humanizing trait, and there's no real way to "personify" the trait "Black". It's just a color. Again, what is our purpose? Our purpose is to capture a set of traits that allow us to observe the real you.

The only way a color can be humanized or personified is if we build in a stereotype. If I call you "black", it must be capturing...maybe your taste in music? Maybe you're aggressive? Maybe you laugh really loud?

Mathematically, if you want to go back to the equation above, you are forcing me to come up with some element in a loading matrix/function z that somehow translates the underlying x = you into this noisy trait, y[i] = "Black". You are assuming such a function even exists. If it doesn't exist, I have to force it to exist in order for the equation to be tractable in my head.

By referring to someone's skin color as a descriptor, you are effectively forcing the person you're talking to to either dehumanize the target of conversation (i.e. a "color" is not human), or you are forcing them to substitute a usually lazy stereotype into the description, just so they mentally render the picture you're painting.

Instead, why not use nationality? Nationality and culture is a clearly human trait: there's no physical process that creates culture in the way that there's physical processes that create color. Culture is human, nationality is human. Moreover, you can parse out the traits you're trying to capture: maybe you're a kid from inner city Chicago who had a hard life and grew up in the projects, but whose family supported him and who managed to claw his way out to go to school out in New York to become an artist. What you've done there is given me a set of traits that actually humanizes you rather than just saying "he's a black kid from inner city Chicago who's at school in New York", which forces me to impute (another math term, sorry) traits into the word "Black" to get a picture.

But wait, there's more! If the person follows up and explicitly asks you their racial background, you are still talking about a human being. This is where the other comments come into play: mathematically, you're actually trying to capture race | human. So the social construction of words now becomes important; as a society, we don't use the word "yellow" to describe Asians, whereas we certainly use the words "black" or "white" to describe those races. The dehumanizing effect of using color to describe a race goes down as the context assumes more and more humanity. This is why when you're among tight friends, you can totally call your Asian friend "yellow" and everyone will just roll their eyes and he'll probably smack you and you'll go about your day. Or he'll tell you 'dude that's not cool' and you'll say "ah sorry bro" and won't do it again. No harm no foul, but you haven't dehumanized anyone.

So the next time you want to think about describing race, ask yourself: how much humanization have we in this conversation already assumed about the target of conversation? If you've established a great context of being human, then you might consider the social construction of racial descriptors and decide it's OK to say "black" but not OK to say "yellow". If you're in casual conversation, just describe the person as a human being first, establish a prior understanding (for the stats geeks out there, yes I just went full Bayesian on you. Bite me.), and then if they follow up feel free to describe race as you will.

TL;DR: Never, ever forget your goal when describing a person. You're trying to capture the essence of a human being, so always ask yourself whether the description you're giving is humanizing them more or less in this situation. When in doubt, always go for more.