I always play on the largest sized maps, and haven't run into any problems.

Sure, there are issues with stuff dropping miles away from you, but thats kind of neat, and it's not like it REALLY matters since everything is tied to wealth, so it's not like you're gonna be "behind" if you lose out on a colonist or something because they die before you can get to them. Biggest bonus is just having more steel on the map, imo. Means less time before you have to start sending out caravans and stuff. Honestly if there was a way to make the map infinite, like factorio or something (rather than the hexes + worldmap thing) I'd do it in a heartbeat.

Going back to smaller maps just feels wrong now. Every raid is like a knife fight in a broom closet

Only real issue I found with bigger maps is that you sometimes have problems with releasing prisoners or visitors when they're injured. If you get some frail, drug withdrawal missing-a-leg prisoner you can't really release them since they'll pass out and need rescued before they reach the map edge, and same with visitors. If someone from a caravan gets their leg eaten by manhunter tortoises or something, and they've got other movement or conciousness reducing preexisting conditions, you're basically stuck with them if you rescue them. Which means either feeding them forever or just letting them die and taking the rep hit. I suppose you could build a temporary hospital bed at the map edge, drag them to it, and let them leave like that, but that's a lot of effort.