Just change the flat +10 weapon attack perk to 10-15% weapon attack increase and remove the 10% damage bonus on "no chest armor" perk.

With those two changes you instantly nerfed the grey horseman's axe headshot hits by 30% and 20% the felling axe. Because it's percentage based and not flat, higher tier axes remain the same. At the same time you removed the perk where barbarians gets bonuses for not wearing a chest and now they kinda have to, which again costs them movement speed.

You could go even further and completely remove the axe specialization perk, but at that point everyone would just move to fighter, because barb lost the "edge" and fighter can use the horseman's too and has better speed boost, access to heal and also blocks, bows and those throwing axes.

Barbarian is the king of cheapness, because you only need the grey axe for few gold and you even get buffed when you wear no armor. Nerf those 2 aspects of it and people move on. Most likely to ranger spam, because all you really need is a green or blue bow and you kill people with the triple shot skill and you can cheese every PvE aspect of the game.