All white males in the US are privileged, and that's not simplistic. Since this is ELI5, I won't apologize for appearing condescending, but you don't seem to understand what "privileged" means.

"Privilege" means that you are the default, and get to decide what is and is not "normal." "Privilege" does not mean that your life is easy, nor does it mean that you get special treatment.

Think of it this way: if you open your mouth to speak, will someone laugh at your accent? No? Then you are privileged. It doesn't mean that you get special treatment, or that you're rich - it just means that you can open your mouth to speak and people will take you as "normal."

Now apply that to other aspects of your life - are you "average"? Are your clothes considered "normal" in your culture? Is your education "normal"? Your religion? See this checklist for a more accurate explanation of how to look at, think about, and understand your privilege and its limitations. (Because, yes, even white males have limitations to their privilege.)

Black people also have certain kinds of privilege - for example, when discussing minorities, most people think "minority = black," which is why you might see "diversity" represented in TV shows by simply adding more black people. They, too, like white folk, are considered "normal." Black privilege also has its limits.

Neither is white privilege is universal. In Japan, for example, Japanese privilege is prevalent - however, like blacks in America, white people in Japan are considered the "normal" minority. Diversity in Japan = more white people. I certainly do not have white privilege in Japan, but I also get away with a lot more than, say, a Chinese immigrant would.

Bottom line, though, if you are angry about being called "privileged" it means that you do not understand what the term means. It is not an insult or an accusation, but a description of your relationship to your culture. Again, this is ELI5, so I'm not trying to talk down to you, but to point out that, if you researched a little, your question would answer itself.

As for all white males being racist, you are again confusing different definitions of "racist," and more research will answer your question.

just because of their gender and color of their skin

Because it isn't just because of our gender or the color of our skin.

Note: I am a white male from a minority white ethnicity (my people experienced cultural genocide at the hands of other whites) and I currently live as an immigrant in Japan. I also have a large family inheritance from a rich ancestor that has kept my family's finances stable, so I understand privilege and its limitations as well as just about anyone could.