First, even if you get the custom planets to work all except the Homeworld will simply be "systemname(2-5)". The only way I've found around this is to simply go to "Details" bottom right of the colony screen of the planet and simply rename it. You don't have to set homeworld in the customplanets.xml either, just make it "PlanetName=Homeworld" and it'll use whatever you put in the RaceConfig.xml for homeworld.
Second, if you have the Homestar field in the RaceCongfig.xml and the StarSystem name in the CustomPlanets.xml are the same there shouldn't be any problems with not spawning in that system.
Granted no matter what I do, the asteroids always seem to ignore what I put in, but the planets always show the proper size and class.
Just make sure you're either activating the mod correctly, or if you're just editing the default files, that you are editing the right version of it. If you're playing the "Ultimate Edition" you'll want to change the ones in the Twilight folder.
Hope this helps you figure it out. Back to trying to win for me with a horribly overpowered race and stupidly large tech tree.