At the heart of the matter of "white privilege" is the implicit acknowledgement that worries over whites becoming a "minority" means that there are benefits to being in the majority and disadvantages to being in the minority.

I entirely agree with you - there are advantages to being in the majority, and a significant component of "white privilege" simply boils down to being a numerical majority (or, to a lesser extent, plurality), and will transfer to whoever becomes the majority (resp. plurality).

If everyone could consistently acknowledge that then we would be making progress. However, I often see people on the left disingenuously asking "What does it matter if whites become the minority? How will that affect or hurt you in any way?"

Ask yourself why there seems to be a correlation between hostility to the notion of "white privilege" and the reaction to the thought of whites becoming a minority.

Any white person with what we might call a "racial consciousness" [many people will just equate that with 'being a racist' but let's note in passing that non-whites are often expected to have racial consciousness and will not be called racists for that reason] will naturally be opposed to both the notion of white privilege (why? because the way it's used in practice is such as to diminish whites' social capital) and the idea of whites becoming a minority (because becoming a minority means losing social capital).