This only a small part of it.
The much, much more likely reason is that the sheer volume of right wing 'content' on YouTube introduces a massive selection and sampling bias that statistically skews the algorithm in ways that cannot easily be overcome.
Imagine that you have 100 YouTube videos in a particular category. Of those, there are 10 that you're interested in and 90 that you are not. Say the algorithm filters out 70% of the stuff you don't want to see, and manages to find 70% of the stuff that you do want to see. That leaves you with 7 videos you want to see, and 27 videos that you don't. Out of a total 34 videos that you get recommended, a staggering 80% is stuff you don't want to see.
This is further made worse by the algorithm being really bad at telling genuine community content and hatespeech apart, which makes it really easy to target minorities in particular because minorities, as the world implies, do not have the numbers to compete.
Fascists know this.
They deliberately pump out as much content as possible, without regard for truth or quality, because they know that it's not about what they say, but about how often it is heard. Their media strategies are 100% about being the loudest one in the room to drown everyone else out. They've been pumping literal billions of dollars into this in the past decade.
Hate Jewish people? Make more anti-Semitic propaganda than there is genuine Jewish content. Hate black people? Make more racist propaganda than there are black people put out content. Hate LGBTQ folks? Pump out more homophobic vitriol than gay people can keep up with.
In each case, the algorithm will not be able to keep up, and will not be able to tell them apart simply because, again, the skewed volumes introduces bias into the sampling.
This is not a problem that can be fixed by adjusting the algorithm or changing viewing behaviour, because it is neither caused nor impacted by either of those. It is purely the result of volume.
There is, unfortunately, no easy solution for this that doesn't involve extremely aggressive censorship and, ultimately, mass deplatforming of fascists.