I think populations drop for various reasons:

1) Casualties fighting off an invasion,

2) Excessive draw-down to supply colony ships and troop transports,

3) Low approval rate.

IIRC, population growth rate can be increased by selected racial traits (race set-up), anomalies (growth increased x%), certain research topics, and certain built improvements. 

However, the base growth rate is directly tied to approval. I have seen specific #s but am unsure if I recall them correctly.  Increasing tax rates soon lead to lower than 100% approval.  The highest pop increase rate is when approval is 100%, and some races double their growth rate then (Thalian?).  I think 71% is one approval point, and 44% is another.  I can't recall the details, but they are probably be in the wicki

My guess is you pushed up the tax rate and held it there, the approval rate dropped low, and you are in Imperial Gov't and did not get any revolts despite your low approval rate.  Change gov'ts to get the bonuses and you will have revolts and defections at low approval levels, but not at Imperial, IIRC.  You can also lose elections, with various consequences.  In Imperial, your folk just emigrate or die of starvation or despair at the taxes!

Is it identical in DL, DA, and TA?

Actually, they just renounce their citizenship in your empire, according to the manual.  So they still live there, but they don't pay taxes or otherwise do you any good!

I think you probably missed the event where the plaugue hits everyone.... That is the only thing I can think of that would make your homeworld drop from 12 b to 1.25 b without getting attacked aside from having very low approval rating.