I understand why viewers are frustrated by Naofumi constantly having to fight idiots like Motoyasu, but him having to deal with unfair treatment which jades him is the premise of the series which was made clear from the first episode. And Motoyasu, while very stupid, is stupid in ways that are somewhat realistic (don't try to tell me you've never met an idiot in your life) and occur based on the kind of person he is.
He's a White Knight who wants a harem, but also wants to believe that he's a good person who is good to women. Hence his self-righteous attitude, but also his lust and pedophilia. He takes being fair to women to include always believing the woman (Myne) even when there is no evidence. And of course he's not going to trust Naofumi over his companions.
He's basically a satire of your typical harem anime protagonist, just like Ren is a satire of Kirito and Itsuki is a satire of a self-righteous person who doesn't think things through.
Honestly, it's better this way. Naofumi facing constant frustrations, obstacles, and enemies, while frustrating for him, keeps things from being a cakewalk like they are for so many series where the protagonist is overpowered, loved by everyone, etc. Where the series would get boring in my opinion is when everyone likes Naofumi and he gets OP. He's starting to get to that point as more people realize he's a good person and he starts to overcome his initial weakness as the least powerful/knowledgeable of the heroes, but at least he has worked to earn that reputation to redeem himself in the eyes of society instead of starting out that way. And the fact that the experience jaded him makes him a more relatable protagonist IMO because he's not an all-loving messiah, he's a grumpy, cynical dude who finds it hard to trust others but ultimately someone who tries to do the right thing, like a lot of people IRL who've been screwed over at one point or another.