The assumption is that time travel to the past would return to the same absolute point in space. Since the mechanics of time travel are unknown, there is always the possibility that time travel could be anchored to a large nearby gravity source, like the Earth or the Sun.