I believe I was building to many farms on most of my planets.
You should only build one farm on each planet which when fully upgraded in ToA will cap your population at 15B (8B for the initial colony + 7B for the best farm) and 16B on your home world (multiple once you start invading the other races).