One major factor is the male reproductive nature. Men have a more readily regenerative reproductive system, and this is energetically taxing when used. Like women come preloaded with all the eggs they'll ever have and then have cycles later that release the eggs. But men consistently make the reproductive juice in their nuts, and if they release, then the body makes sure to use energy to 'reload' so to speak, and this is one of the top priorities in the body besides survival of the system itself. So men can actually consciously tax their energy using their sexual system, and can do it to a much greater degree than women.

I think I remember hearing something about testosterone kind of being harsher on the system than female reproductive hormones too?

Not to mention, from my observations and personal experience, overall men are far more indulgent than women in all aspects of their life so this could disproportionately tax the system when compared to women.