I have never had that myself, however if it is the taxation than holy cow.
If your approval is too low, you don't get new population. If you don't get new population than you don't have the tax base to maintain your empire. Do you by any chance have persistent money problems?
I don't play on tough because I am too much of a wimp but that seems like a bad place to be. It seems to me the Terrans play best with lots of people and better governments. You get an economic boost with each level of government.
Is this not so at tough as well?