Well, that's potentially very easily solved. Consider the potential natural elements that could lead to a wormhole and the rarity of such elements combining in nature. You could enter a wormhole in the Milky way and end up in a section of the universe that Earth does not know about. Any time travel you've done is negated by sheer distance.

This is only true if you assume the time you travelled to the past is no more than the distance divided by the lightspeed (t < s/c). If it is more, you would have time to just fly back and return to your previous position *before* you left.

And what if there's another wormhole over there? One that works pretty much the same way, also sending you to the past, only to somewhere around where you came from, meaning you effectively timetravelled. If there exists such a thing as a wormhole, there has to be a way to prevent this... or not depending on what view of timetravel you have (there's many different ones).

The answer: "This will probably not happen, because they are rare" is not satisfying to a physicist, because anything that *may* happen according to the laws of physics, probably will happen somewhere, because the universe is so large.

That said: With what we know, most physicists will consider wormholes science fiction only. They won't say wormholes don't exist, but you won't see any galactic surveys looking for them.

As for observers seeing travellers traveling into the past: I don't think this has ever been worked out right (mathermatically), because FTL travel is not considered to be possible, so it's kind of up to your own imagination how it will look/happen. I think there's even multiple interpretations to FTL travel and time travel if you sit down and think about it... something I'm not gonna try right now:P