Because "yellow" as a descriptive term had nothing to do with the actual colour of people’s skin and everything to do with white persons racially mapping out the world in a way that benefited them and could justify their actions in terms of who is civilized, cultured, literate, and obedient and who is not. Prior to the 19th century, Asian people were considered as white as white people. But they had their own global interests and desires that were not aligned with white people. For that reason, white people changed the narrative by racializing them.

Black and White are not the best terms, but IMO they are better than terms like "Caucasian" or "African American" which both carry strong connotations about where someone may be from - connotations that are frequently very wrong. That said, everyone has their own opinions on how they identify themselves. It's important to listen to how they refer to themselves and follow their lead.