In my experience, more is better/easier. There's a better likelihood that one or two of them may challenge you, but there is also a better likelyhood of getting allies to fight with/for you. You can play some of the ai's off against each other etc., and if you're the kind of person who gets a kick out of tech whoring, more ai's causes that aspect of the game to be much easier. I find that actually lowering the number of ai's to sucker...err... to challenge you, actually ramps up the difficulty level a bit. Toughest games I play nowadays are with no more than 3 other races.

Just o n e t u r n m o r e .. before I go to bed....well maybe one more, just one more,.....

when I looked at the clock it was 2 and a half hours later in the blink of an eye!

That is the only problem with this game, it is addictive with the "just one more turn" attitude.
I once stayed up until midnight playing the game because I didn't realise what the time was, I find the best way to stop playing is to finish a game because then most people automatically look at a clock to see what the time is.

As for the number of opponents, I'd agree with what has already been said.