I've never tried using guard or sentry and while in a perfect world I agree that it would be nice for these functions to work as expected I don't find this lack to be any great difficulty because leaving a single even unarmed ship in orbit will stop a troop transport from attacking a planet.

As to why this function may not appear to work perhaps there's a particular distance at which this function gets activated. If the opponents troop transport ends it's previous turn outside this distance but has enough speed to reach the planet in it's next turn then the behavior you've described is perfectly understandable. You could also simply scan for enemy troop transports yourself and attack those within range of your planets. That's what I do anyway.

You can also use Evil Stormbringer's patented method of merely putting a tiny hull in orbit around each planet. This ship has no engines and no weapons, it's merely a tiny hull with nothing else on it, but it stops an unarmed transport from attacking the planet. It has zero maintenance and can be built in one turn on any planet even with no industry other than the initial colony building. It's a great idea and a very useful tactic.