If you believe it is justified to restrict the freedom of speech for alt-right because you think they are motivated by hate, you should also accept your own speech being restricted by others on the grounds that they think you are motivated by hate.

This seems to only follow if one is a relativist or otherwise believes that everyone's truth is equally valid.

Freedom of speech isn't about allowing speech that you agree with, it is about allowing speech that you viciously disagree with.

The point of free speech is that we're better off letting competing ideas, well, compete, but this assumes that different ideas actually have value. We're better off letting people argue about the merits of different taxation systems and different philosophies related to government spending, or the role of religion in society. We're not better off letting nazis drum up support for an anti-liberal revolution, because liberal democracy is preferable to fascism.

Whether liberal democracy is preferable to socialism, communism, or what have you is an entirely different question, and restricting the rights of fascists should not be compared to restricting the rights of socialists, because fascists and socialists are not identical.