Good questions. I think a great deal depends on the distance and quantity of the asteroids, as well as the math, which isn't available in the Galactopedia (you have to check in game to see how many sheilds they are producing). As asteroids get further away, their value diminishes greatly. But if you have a bunch near your planet, then when you have some breathing space, kick out a miner or two.
At the colony rush stage, I will usually have my miner build 2-3 mines before I upgrade it to a colony ship. Other than that I am with you, one rarely has time to spit out spaceminers and when you do, you are usually already winning. If there are no galactic resources left to mine, and I am not ready to go to war, then I will usually produce a couple space miners. That being said, think about the math (and please correct where I am wrong):
A manufacturing center produces 10 shields and takes, later in the game, oh, 10 turns to build. A level II mine spits out, err, 10 shields and also takes 10. But, the number of space mines are limited only by the number of asteroids, which, depending on your settings, can be very numerous. I may only be able to build 5 manufacturing centers on the planet but could be surrounded by a dozen asteroids. In this case, I may get more production from the asteriods than the manufacturing centers. So from the math, as you can see, they can be very competitive with production planetary improvements. This is assuming that the mines are close by. If they are two squares away you usually get 4/5 shields for Level I, which is very helpful. Why is this worse/different than a Xeno Factory, which produces only 6 mp? You could make the reverse argument, why bother with xeno factories when I could just build space mines. They take 5 turns to get 4/5 MPs? Great deal.

There is another aspect to the math: space mining is expensive to research and other civs will not trade it to you under any circumstances. I think it is actually a bug. I remember trying to trade my home planet for space mining and the AI civ told me that "Space Mining is too important to our strategic goals to trade at this time." So, I'll usually only have time to research the other levels when I'm producing so much research that I can knock it out in 1-2 turns. This usually means I'm winning anyway. The argument could be made, though, that instead of researching manufacturing centers, you could research SM and then trade for MC's when the AI researches it (they will trade MC's, for a lot).
Therefore, one thing to try is to research space mining and offset this by trading for the production improvements, and have space miners continuously a-building. Once again, provided that the asteroids are close by.
Here are my questions:
1) Do the MP's from space mines contribute to your base production or are they added to your production after bonuses are applied? If the latter is the case, then much of the reasoning here is weakened.
2) Do space mines have maintenance like PI's?
3) I have noticed in the campaigns, against the Dread Lords, that they tend to ignore your ships if they are hiding in asteroid fields. I have observed this time and again, although it is not 100% of the time. There is simply a drastic reduction in the number of times the Dread Lords go after your ships if they are hidden. Is this an illusion or is it part of the AI?