I don't have any personal experience with big game developers, but I'd guess that physics are probably offloaded the most. It turns out that writing a 3D physics engine capable of working in realtime is pretty darned tricky; there are all sorts of optimizations that are more or less required and loads of edge cases...it looks like a nightmare.

And happily, physics engines are very easy to separate from every other part of an engine; it's completely agnostic to input and graphics, and the core math is generic enough that the libraries can provide interfaces that allow developers to write their own collision objects for custom features without re-implementing that whole headache.

Graphics isn't actually that bad; like you said, thousands of man-hours have been spent refining the algorithms; most of them are actually from decades ago, it's just that we've only recently gotten hardware powerful enough to run them in real time. So writing a simple 3D engine these days is probably do-able, even for small teams.

I'll bet most people use off-the-shelf libraries for stuff like reading mesh/image/data/etc file formats too, though.