Last year I was on a vacation that ended up costing $30k, it was supposed to be a little under $3k. It involved having to pay SAR, chartering a plane, new clothes, new meds (which insurance wouldn't cover), and changing plane tickets home. Luckily health insurance did cover the hospital visits as in-network or it could have been much worse.
It's definitely not the first time a trip I've been on has gone extremely sideways and costing $1000s (or $10000s) more than expected.
While I do get travel insurance, there are many cases it won't cover that can end up really costing.
Personally, I prefer to have room in my budget for those cases. I don't mind working longer to account for the possibility.