Thats odd I have a LG lg 34gn850-b(144hz 10bit normal max 160hz 8bit) and can set 10-bit 154Hz and other custom refresh rates no issues VRR works fine etc.

What happened did it artifact/error msg/blank or just not show up in the windows refresh rate toggle?

If the latter try 120Hz & 80Hz etc it's buggy once one custom refresh rate option shows in the windows advanced display settings refresh rate toggle and you set it that custom refresh rate the refresh rate toggle in Nvidia control panel changes and now shows all the custom refresh rates and you can switch 144Hz 120 etc there. For me the 120Hz was the custom refresh that showed up in the windows toggle.

Sorry if my explanation was too confusing.