We have a vacation budget of about $6k a year, but this does not include airline miles that we use (which we collect from work trips and personal trips we pay out of pocket).
This is how we save money:
Don't pay for petsitting. Granted, we only have a cat. But we've had good luck with people from TrustedHousesitters.com who want to stay at our place in return for petsitting. Annual membership was $80/year, much cheaper than paying someone. Some of them have even driven us to/from the airport.
Stay at AirBnB. Unless it's a really cheap country, we stay at AirBnB. This is usually cheaper and more unique. Plus, you usually get a kitchen and a fridge to drastically cut down food costs. Also, snack instead of eating full meals. (Yes, we've had a couple of not-stellar AirBnB experiences. But overall, we enjoy the variability.)
If we're traveling, we shift the prorated food budget towards "travel". But it usually doesn't cover the entirety of food while traveling. Still, it makes our budget stretch further.
It also sounds like you fly a lot? Don't do that. That's expensive. We try to plan 3 "major" vacations a year: 1 for traveling to foreign countries, 1 for visiting family (mine is local), and 1 for fun/nature/adventure.
3-4 weeks on an international trip (either a cheap country, or travel with work, so partially paid for by work, around $4500 plus food budget)
1 week to visit my husband's family (Thanksgiving or xmas, we try to buy tickets WAY ahead of time, or just drive down, and it only costs us $500 or so)
1-2 weeks of fun/nature/adventure. We've gone backpacking before. This year we might go sailing. These are things that might cost $1000 or $2000 in gear/rental, but if you don't need plane tickets and pack all your own food and don't need to pay for a hotel, it becomes affordable.
Long weekends, we usually do staycations, or day trips.
We also take other work trips. (We both work.) But if it's not somewhere exciting, then it's not worth it for the other person to take time off, arrange for a petsitter, etc. I think it's good to have alone time (especially after you've just spent every single minute together for a month straight).