As far as I know you can't mod the actual AI modules themselves, but you can make significant improvements to the TOA AIs in two ways:
- AI personalities 7 (Drengin, Yor, Korath) and 10 (Torians, Iconians) are bugged, but AI personalities 8 and 11 are not. Switching the bugged AIs to one of those personalities will cause a significant improvement in their behavior (although I would keep the Korath on 7, because it's the only way they will use spore ships. With the right settings, they can expand through conquest instead of aggression, so it works).
- Many AI tech trees had techs set to the wrong priority value and/or categories. Fixing this makes the AI make much better use of its techs. This takes a lot of work, but at least one mod has been released that does it for you.
Fixing both these problems makes for a greatly improved AI.