Factors such as intelligence, attractiveness, height, wealth, even hair color all have noticeable affects on one's life. I do not see why there shouldn't be an intelligence privilege or an attractiveness privilege in the same vein as white privilege. All three of these kinds of privileges give you noticeable advantages, while still being uncontrollable and (largely) determined at birth. If we are going to say that white privilege is not ideal and we should seek to eliminate it, then it seems illogical to not say the same about a number of other privileges.

I'm from the USA, so I'm going to answer this question from a US perspective. For Americans, white privilege is worse than beauty privilege because white privilege is connected to a long history of evil. There was no trans-atlantic trade for beautiful people. There was no Jim Crow against intelligence. People like Emmett Till and MLK didn't die because of their hair color.

As America evolved as a nation, it adopted anti-racism as one of its hallmarks. It's not just about unfair advantages, it's about an extremely complex and brutal history that needs to be corrected. That's why Martin Luther King day is a national holiday. His accomplishments against racism are so important to us as a nation that everyone needs a day off just to celebrate them. That is why, at least from an American perspective, white privilege is a more important issue than beauty privilege, etc.