I like to consider the idea that time travel may be possible through a sort of gate, and that no one has traveled to the past so far because the earliest you can travel is to the creation of the first gate. But once the first is built, the floodgate will be opened and people from all times will be appearing through it.

In terms of only traveling via observation, there is an excellent short story by Asimov about this very thing.