I found the combat of both frustrating in a way that Yakuza 5 isn't. The combat system is pretty similar to 0/K1 and doesn't feel especially bad to play, but the enemies are annoying. 3 has them block way too often, and 4 has certain enemy types that are quite irritating, with 2 bosses in particular really annoying me a lot (I beat them without dying, it just didn't feel good fighting them). 4 generally fares better than 3 outside of those fights though.

I would say if you've been playing on normal and easily mowing down your enemies on 0/K1/K2, you'll probably want to play on a low difficulty in 3/4 to minimize frustration and enjoy the story/activities. If you've been playing on hard and make full use of your move sets then you can probably learn to deal with the enemies 3 and 4 just fine.

Adjusting to the graphics and engine generally wasn't that bad though honestly and I'm still happy to have experienced them despite having a miserable time with some bosses in 4. It helps to know the story and characters of 3 and 4 for the sake of 5 too.