I've read this thread and made some investigation on my end.

Mac + Chrome + keyboard -> no lag Mac + Chrome + stadia controller -> no lag TV with Chromecast ultra 5g and stadia controller: lag

Everything at 4k resolution.

I made a tour of my TV settings and discovered that it's all about the image type of the TV. I've had the same problem with other games and switched the image mode to "game mode" but apparently it reset (probably because I've used different video inputs other than CCU on HDMI1). Long story short, I've switched the TV on game mode again and the input lag on skill checks is essentially gone, I'm so happy since all my friends mocked me a lot thinking I was shit with skill checks. Which I am, but at least now it's all my fault!