Keep in mind some buildings cannot be removed or "upgraded"... you'll see that as you take over enemy worlds.Also, use it to your advantage.
Using it to your advantage includes the following behaviour:
You can't queue up another unique building. However, if the planet you conquered already has it built, you can keep it. More, if the planet you conquered just has the building in its 'to build' list, you can still build it!
That means by carefully choosing target planets for invasion (using tactics like information warfare that have minimal collateral damage) and having some luck with others, you can have multiple tech or manufacturing capitals in your empire!