do we have the right to defend ourselves from hate speech?

No we do not have the right to physically harm other people over words unless those words are a direct IMMEDIATE threat.

I find it hard to imagine a Nazi getting punched unless they deserve it by actually being a Nazi.

Leftist have shown that they are incapable of having clarity on who qualifies as a nazi or not. John Macain was labeled a nazi when he ran for President. Ben Shapiro (a strict practicing jew) has been labeled such, Steven Crowder had had to get his lawyer involved because a leftist professor was spreading fake images of him with nazi tattoos on the internet & on his campus.