When he discovered SS3, he was dead and had no stamina problems. When he was brought back for a day, it shortened his 24h lifespan. When he was completely revived by Old Kaioshin, after some time, he couldn't hold it, and not even energy gathering helped. People are hung up on statements made by Toriyama himself:

Goku endlessly keeps getting stronger, with Super Saiyan 3 in the manga and Super Saiyan 4 in the anime; does Super Saiyan keep getting limitlessly stronger too? Might we eventually see things such as a Super Saiyan 5…?! Super Saiyan 2 and 3 are nothing more than powered-up variations of Super Saiyan. After the fight with Beerus, Goku realized that mastering his normal state and Super Saiyan would raise his level more and sap less strength, so I think he probably won’t become Super Saiyan 2 or 3 any more.

SS3 "saps strength".