I have two cases here.
1) A standard transport that I sent to capture another race's planet could travel anywhere around the enemy's planet but could not capture it. The ship started from a planet in space I own and went to a star in space the enemy owns in which I have a colonized planet. I could not attack the enemy's planet but I could add population to my planet (in the same star system). Attacking the enemy's planet gets the "out of range" message.
2) I have a planet in an system owned by another race (they have two planets, I have one) where I built a standard colony ship. When I try to colonize another planet in the same system it gets the "out of range" message.