I think they are perfectly in their right to ruin a streamers stream. Streaming in a public area is a total Karen move.

Look at it this way, say IRL you and your friends are hanging out in a park, or in a coffee shop. Someone you don't know pulls out their cellphone and starts streaming to YouTube/Facebook/Twitch, they start shoving that cellphone in your face, they start yelling racial slurs, start making fun of the way you look, lean into you and start taking selfies with you, start touching your face or sitting on you. Then they post that video on social media and share it with everyone they know. That's an asshole move IRL. And it's an asshole move in VRchat. The only problem is, in VRchat you can't know the identity of the asshole, you can't beat the shit out of the asshole, and if you mute or block the asshole they can still stream you.