When you move off pavement, you instantly loose the ability to grind out climbs, keeping traction becomes a huge battle, keeping the pedals chugging at the same time is difficult.

single speed can definitely be limiting, but in saying that here a trip report of someone who traveled the world on one gear

You essentially completely either forefit the ability to go fast at all, or the ability to climb any sort of significant hill. It seems like most people do it as a challenge, not for functional reasons.

And I'll just mention dingle and tringle speed setups as another possible option, been toying with doing a vintage rig build that's 3x1 front mech only, not at all practical but interesting!