Let's say I play the Terrans in one game and choose various abilities for them. After that, I start a new game and choose a different race. Will the AI Terrans use the abilities I chose in the previous game
Yes they will.
or is there a default set of abilities for each race that the AI will use?
No there is not.
If I've never played a race and there are ability points unspent, the AI will, I hope, spend them, right?
And lastly, no they will not.
If you want the AI races to use ability points you need to go in and set up each of those races and their abilities prior to playing a game. And of course, those settings will carry over to the next game unless you load the default value for those races.