That's the case of Time Travel, You cannot travel in the Past. Not only does it create Paradox,

But also the very fact that a single affinity cannot be at the same place.The Same matter cannot be in the same place provided a significant time has passed such that each and every atom in your body has been replaced.

What I am trying to say is,

Same Entities cannot be at the same place in the same time(in the past) let alone the same place until and unless and hypothetically, The Atoms of the bodies havent been replaced completely.

You cannot go to the past because it creates a Paradox. Your mere existence could change the timeline into something very different. Think about it this way, Somehow you did manage to get to the past. The Moment you took a breathe, the existing particles changed there places which increased so much that in a significant time period it became the reason of a huge cyclone or a hurricane which caused devastation and due to that some people died, people which were significant in the timeline for something etc etc, you get my point. And If Time Travel were to get True, the Theory of Multiverse would automatically become true.