The bottom line. If you can use a tile to increase population by 56 % its worth it, otherwise its not. Does anyone else have a different opinion?
Seems logical, however it's not a very practical calculation that one can make primarily because one can't simply use just one tile to increase population it usually requires some combination of farms *and* morale buildings.
However I do think you're on the right track.
The way that I look at the problem is to determine whether the loss of income from fewer stockmarkets is worth the increase in income due to a higher population for a given number of tiles that must be converted from stockmarkets to farms/morale buildings. The important part that most people neglect in these kinds of calculations is the benefit of food/approval bonus tiles and it's really these tiles that make all the difference in the world as to whether it's worthwhile to go for the higher population or not.
To simplify the problem further it's reasonable to consider only two different populations that one is willing to support on their non-civilization capital planets. For DA the easiest thing to consider are planets with one Advanced Farm for a total population of 13B or planets with two Advanced Farms for a total population of 20B.
Basically for "normal" levels of racial morale ability plus "normal" numbers of morale resources to mine in a gigantic galaxy you can usually maintain "acceptable" levels of morale on planets with a population of 13B with no morale buildings at all. Also in general under similar gigantic galaxy conditions it usually requires 2 VRC's to maintain "acceptable" levels of morale on planets with a population of 20B. Note that I'm not requiring the levels of approval in both conditions be the same only that they both be in some acceptable range. However it does assume that in both cases your tax rate is pretty much maxed at 79% or 80%.
If the above is all true then your tradeoff is clear. Maintaining 13B requires 1 tile (1 Advanced Farm) and maintaining 20B requires 4 tiles (2 Advanced Farms and 2 VRC's) and so a pop of 20B costs you a net of 3 tiles that cannot be used for stockmarkets. The question merely becomes is the 75% income bonus of 3 stockmarkets more or less than the income increase of the pop going from 13B to 20B.
If the above is not quite true then you simply need to figure out what the tradeoff is in your particular case. All you need to know is how many tiles you need to give up for food/morale buildings and that will allow you to figure out at what pop each planet has the maximal income.
Actually the comparison really depends on how much econ bonus is left on the planet *after* taking away 3 tiles for farms/morale buildings. Clearly if you have 23 stockmarkets on a planet for a total economic bonus of 575% then taking 3 stockmarkets away only reduces the planets total economic bonus to 500% and so the income increase due to a population increase will be worth 5 times it's base increase. Whereas if a planet only has 3 stockmarkets which gives a 75% income bonus to the lower pop condition then that needs to be compared to the planet at the higher pop with *no* income bonus. Clearly these cases will result in two different outcomes.
Note that it matters how you build out your planets. If you go with an "income only" strategy with a maximum number of stockmarkets on each planet then you're more likely to be able to justify a higher population than if you "waste" tiles by having factories and/or research buildings on them.
With all of the above in mind now take a look at the Income formula from the Wiki article, https://www.galciv.wikia.com/wiki/Taxhttps://www.galciv.wikia.com/wiki/Tax.
constant * sqrt(population_in_billions) * tax_rate * (1 + (sum from buildings on planet)) * (1 + (sum from racial bonuses/maluses) )
Note that since we're comparing income at one pop with one set of buildings to income at another pop with a different set of buildings the constant, the tax rate and your racial bonuses/maluses are all constant and can be ignored in the above equation and so for comparison purposes the equation simply becomes the following.
Income at 13B = sqrt(13) * (1 + (N+3)*0.25) versus
Income at 20B = sqrt(20) * (1 + N*0.25) where N is the number tiles left for stockmarkets in the high pop case.
If the income is equal then I prefer the solution with the higher pop and so in general I'm willing to accept slightly less income in the high pop case.
So if you plop those equations into excel you get a graph that's fairly "flat" and if you really want the max income without regard to population then you find that you need N ≥ 9 before it's worthwhile to be at the higher pop. However if you're like me and are willing to accept the higher pop as long as it's at least within 5% of the lower pop income then that lowers N significantly to in this case 6.
However you're not quite done yet because even if you're a purist and insist on N ≥ 9 there are many cases where the correct value of N is *much* lower which are when there are bonus tiles on the planet. If a planet has at least one 100% food bonus tile (note 300% food bonus tiles are useless) *and* one moral bonus tile then instead of costing 3 tiles to go from 13B to 20B you only need 1 tile. This changes the graph significantly since the N+3 in the first equation becomes N+1, and under these conditions it's always worthwhile to give up the one tile and be at 20B for all N ≥ 0.
Also if a planet has only one of either a food bonus tile *or* a morale bonus tile then you need to give up only 2 tiles to maintain a pop of 20B and in this case it's always worthwhile for N ≥ 5 (or if you're willing to accept 5% less income for the higher pop then N ≥ 3).
So basically whether or not it's worthwhile to have a lower or higher pop on a planet depends upon how many tiles you have to give up to maintain that higher pop as well as whether or not you're willing to give up a small amount of income to achieve that higher pop as well as how many tiles you "waste" with non economic buildings.
This is a practical calculation that anyone can make for any particular game and play style and so at least you can *know* that it's economically worthwhile or not to maintain any particular population level that you wish.