If challenging becomes too easy, it's probably time to move on to tough + add a few more opponents. In my experience playing against only 1 AI is pretty straightforward, but a few more races in the game add to the difficulty:
- more constraints in the colony rush
- less chances of grabbing those precious resources
- more chances of getting declared war on, especially if they start ganging up on you
- more chances of falling behind in military / expansion / research / ...
And yes, changing your own race makes a huge difference!