According to The Ancient One, the branching timelines would be eliminated so long as the stones were returned at the exact times they were taken. They also very clearly stated that their timeline couldn't be affected by their time travel. Thus, nothing they did in the past should have affected anything so long as they returned the stone.

So the issue isn't "that should have messed up the timeline" but rather "why were they sneaking around in some of those places since they couldn't change anything anyways?" It still makes sense for them to sneak around in NYC so they don't have to fight themselves, but they really didn't need to sneak around Asgard, space, or the SHIELD/Hydra base