The Ancient One says that the Infinity Stones create the flow of time (Time Stone) and that removing it would remove their ability to defend against the forces of darkness (Dormammu, who is a timeless creature). I am not sure what exactly Banner is referring to when he says "erase" but we know that he isn't referring to erasing Alternate 2012 because the Ancient One was trying to protect it and Alternate 2012 Loki is still out there and is receiving a show.

None of the other realities can be gone either:

Thanos, Gamora and all of his forces are gone from Alternate 2014.

Frigga, Jane Foster and the Asgardian soldiers were aware of the presence of Thor and Rocket in Alternate 2013.

The Avengers never returned the baseball glove from Hawkeye's test run.

Howard is now aware of Howard Potts (Tony) and the Tesseract container is broken in Alternate 1970.