If you just change the difficulty on an existing save with maxed out weapon and armor and all the charms then I could see it be easy because you have the high damage, hp and most likely all the quality of life perks unlocked. I would definitely start fresh because at the start archers can be a real pain in the ass as it's a 1 shot, and for a while you get one shot by the basic swordsmen and lack the rolls/perfect parry etc to take them out easily.

Which is why I've found myself having to play defensively and use the ghost stuff more because you can't really afford to take a hit so you may dodge or parry one but as you counter attack another, or even 2 may be attacking you from the other side so you have to react to that.