There's another reason, which I only started to understand recently:

Systemic racism means that, today, a disproportionate amount of people of color live in poverty. Because of budgeting (particularly in the US, but here too) schools in impoverished areas tend not to be as good as schools in other areas. Combine this with the other aspects of living in poverty and an awful lot of people of color will look objectively worse on paper.

Affirmative action policies (quotas) try to adjust against that systemic bias. They try to make it so that people of color aren't caught in a cycle of poverty by giving them a chance to get out that they wouldn't otherwise get. Of course, it doesn't work that way all the time—if you're hiring and it comes down to a white man who went to a prestigious school, a black man who went to a prestigious school, and a black man who graduated from a school you've never heard of, you probably won't pick the latter—but it does help. Just because on the surface it seems unfair doesn't mean it really is; it's actually pretty important in helping combat racism and poverty.

Everyone deserves a chance. This sort of policy exists to give that chance to people who were denied it growing up. It may not be the best way to handle it (I certainly think there are better ways), but it's better than doing nothing.