The manga is very popular in Japan but the anime adaptation faced some criticism. Dumb shit like "it's too cinematic", "CG Bad", and apparently people don't like Denji's VA.

When I first heard about it I wrote it off as a small group of loud people, but the lacking Blu Ray sales suggest that response isn't super positive generally. Blu Ray's actually a very important metric of success for Japanese audiences.

As for the animation taking a hit between seasons, it does happen. Maybe "much lower" was too hyperbolic. I'm not talking about a One Punch Man style situation where it just goes to complete shit, but quality drops on shows aren't unheard of. My Hero Accademia's a good example, albeit for different reasons. While there are still some brilliantly animated segments in every season, there are clearly a lot more shortcuts taken during Seasons 3-5, when they were also working on the films.