As somebody that's been in retail, a custodian, and now an office worker, I can say there are equal trade-offs.

What I lost in people yelling in my face/doing physical labor, I gained in people yelling in my ear on the phone/gained weight from never being on my feet.

My girlfriend is a server/bartender, and her (and her large peer pool's) social skills are top-notch while her days fly by. I don't work with anybody who isn't a parent nor within 20 years of my age, and my days drag by sooo slooow.

I prefer the office job because it looks better on my resume, but that's about it. Driving from one box via another box to another box so I can stare at a box all day isn't always the best.