There is a difference between voicing your opinion and harassing and threatening an individual.

What is that difference? It seems to me that someone who is posting agreement with the Nazis is just threatening a larger number of individuals. You also haven't explained why protecting an individual from threats or harassment does not violate free speech. If I wanted to, for example, call your employer and put up billboards by your work that says "skankhunt_40 is a convicted child molester", why should anyone be able to stop me? After all... you just said that free speech would let me voice my opinion. Maybe that is in fact my opinion.

There is no good reason to create exceptions for free speech that protect an individual, but not groups. The only difference between those is numbers.

You are not directly threatening anyone

Tell that to the people in the US who use their free speech to advocate for ethnic cleansing. That is a direct threat.

you are not physically harming anyone

The same could be said of false advertisement. What physical harm have I caused you if I sell water and tell you it's a tonic that causes weight loss?

you are not forcing anyone else to do those things

So? I'm not forcing anyone to sign up for a ponzi scheme. Those are still illegal to promote.

You are simply voicing your, although controversial, opinion.

Which also applies to my earlier example. Opinion could be used to justify literally any type of speech that the US bans. Harrassment? Just stating my opinion. Threats? Just stating my opinion, I wasn't actually going to hit him. Fraud? In my opinion, that car is perfectly functional with a broken axle. False advertising? In my opinion, this drug DOES help cure cancer.

Hate speech is as logical as any other limit on free speech. We have always limited speech based on causing direct harm to others. Hate speech is a method to intimidate, harass and harm. Why is that illegal to do to an individual, but perfectly legal to do to large groups of them? It's absurd. We don't charge people LESS severely because they have more victims.