It takes a tremendous amount of energy just to perform the "easy" trip from Earth orbit back to the surface with your cargo intact

Nobody was talking about going from Earth orbit to anywhere else. Now I understand why you've been struggling with this so much.

The physics just don't work out to there being any advantage to moving your agriculture offworld.

It absolutely does, if you reverse the shipper and receiver in your example. In fact, going from an orbital farm around Earth to the surface of Earth would require vastly less energy than shipping from any point on Earth to any other point on Earth. Since you understand orbital mechanics and the physics involved here, I'll let you figure out why.