You pretty much have to "build" spies even if you don't want to use them, as you'll need them to counter/nullify spies the AIs use against you. It's not really that big a deal though.

If you don't get the agent mega-event, they are a reasonably benign part of the game. If you have an opponent with a weak economy, like if they bankrupted themselves in the colony rush, you can place an agent and it should last a while. If your opponent has a strong economy and/or it is late in the game, your agents won't last long and they become mostly irrelevant to all concerned. You will still need to build some to nullify enemy agents, but it is a zero sum game.

Agents aren't that big of a deal, I mostly ignore them (unless I've got one on one of my planets). Half the time I end up having a huge stable of agents because I seldom place them.

I had a problem with agents once during the campaign, it seems that I couldn't research counter espi. tech (which allows you to build a building that makes your planet immune to agents), and the other 3 races must have all been using thier agents against me for the number of them that I had over my planets.

Quite annoying, to say the least.

It didn't help them in the end, I still won