Before I invaded the Caranoid planet I had placed several spies on their planet, this was to ruin any attempt to build starships.

After the invasion it seems that my spies were still trying to sabotage the planet, and so I was receiving the penalties..from my own spies!

Thankfully removing them VIA the espionage menu proved quite easy.

SUMMARY: When a planet is invaded and the invader still has spies on the planet, said spies will still be in effect.

RECCOMENDATION: Create code that sends the spies to the pool of avaliable agents after a successful invasion.

EDIT: Seems the same applies with cultural conquest.