Is the manual really correct? Does a colony with 6bn people in it really have the same number of births as one with 100,000 in it? Only a set number of people per planet can have sex regardless of population?
This really puts pressure on you to produce lots of colonies instead of trying to develop the ones you have because that's the only way to boost population growth (at least initially).