So here's the thing: if disproportionate outcomes happened isotropically - that is, approximately the same way and to the same extent in every direction about equally as often - then it wouldn't necessarily indicate racism. Like in your example, if the top four finishers of a race were black and the last four were white, that's something that could just happen.

But one of the many problems is that we can measure these things and it's not even close to isotropic. Disproportionate outcomes favor white people to a statistically overwhelming extent; you have to really cherry-pick for examples where white people are selected against. Even if you try to argue that the individual disproportionate outcomes aren't racist, there's still a pattern going on when you consider all disproportionate outcomes.

This is kind of like how people don't understand the point behind the Bechdel test. The point isn't that a movie is only good if two named women talk to each other about a subject other than a man. The point is that very few movies clear that bar, and meanwhile, almost every movie clears the inverse test (two named men talking to each other about a subject other than a woman).