The "privileged" label is just a way of looking at the same age-old problem (racism) from a different perspective. From your perspective, your life is normal and people who experience racism have it bad (which is valid and true). However, from the flip side, for example for a black person who has experienced racism their entire life - that's what they know and the life of a middle-class white person does seem "privileged" (also valid and true).

The "privileged" label also applies when people who have no fear of consequences commit acts that would get a minority in a lot more trouble. Recent events have shown that, for example, black people are held to completely different standards when it comes to how to behave at a traffic stop (e.g. you may get shot for doing nothing!), while someone else could probably get away with mouthing off to the police officer and at worst get a ticket.

The label can apply here too. Can you image if a bunch of Muslim kids from Western held up a racist banner during homecoming? Do you think they would be treated in the same slap-on-the-wrist way?