I feel like having more items in the fridge to take up more space would save energy in the long run.

If I recall correctly, energy is lost from the cold air escaping every time the fridge door is opened. The fridge thermometer will notice the warmed air and run more cold air through until the air is cold enough again.

So less space for air means less cold air loss, which could mean less runtime of the fridge. Also, more items in the fridge means there is more ability to retain coldness. So when the fridge turns back on to cool the air again, the cold items are also cooling the air as well speeding up the process.

That's my speculation anyway.