It's not racist to base merit on criteria other than the SATs and GPA. Race in itself is not good criteria either, but people may see a correlation between unacknowledged merits and race and assume that the correlation is direct.
Look at it this way: Harvard is a brand as much as it is a school, at this point, and in protecting their brand they want to have high impact applicants. Having a number of people who are very skilled but will all enter the same field is not enough. They want applicants to be seen at the top of the totem pole in many walks of life, rather than absolutely dominating a few specific fields.
This means that they want people who are going to try to change the world in many different walks of life. Harvard isn't interested in people who want to enter a field just to have a good, safe career. They want people pushing the envelope and making history. Minorities tend to have a much greater drive to do that, while white applicants may not. In fact by focusing so much on GPA and SATs it sort of proves they're missing the point and aren't qualified for these schools due to their mentality.
You need to meet a minimum SAT and GPA level to be looked at in the first place. Harvard is not grabbing black people with 900s just for the sake of having black people. However, their overall criteria is not directly tied to SATs and GPA.
Contrast with MIT. While not an ivy, and perhaps because of that, MIT generally does want to dominate a few specific fields. Unsurprisingly they select the absolute best for those fields, which tends to disproportionately be asian and white people. It's why MIT has an Asian admissions rate 30% higher than Harvard. They still, however, have minority representation likely for the reasons above, though in significantly lesser degrees.
TLDR: People assume that colleges aren't choosing meritocratically, but merit is not limited to GPA and SAT scores. One's motivations, goals, ambitions, and potential impact on society are all merits equal to or more important than GPA and SAT scores.