That sucks. I haven't actually had this happen yet, but I'll be on the look out for it now. I really think they need to do something about the "We know what you are doing" factor because most of the time it's just plain wrong. They are also quite sensitive about it, yet have no trouble leaving their own fleets and transports strewn all through your territory if the target they were headed for makes peace with them or is defeated.
In my current game, some minor race (who's planet I haven't even found yet) declared war on me because they thought I was going to attack them. The only thing I can think of was that they thought my troop transports passing by the influence starbases they built in
my territory were somehow a threat. Sure it doesn't matter with a minor, actually it gave me an excuse to clear out their starbases they kept putting up in my territory, but it clearly demonstrates this problem with the AI.
I almost always have the "We know what you are doing" from everyone nearby
except the race I'm going to attack because I keep my transports away from potential enemy ships that could be built, launched and attack in a single turn (although I'm not sure yet if the AI is even capable of doing so).
Edit: Another thing I was going to mention about these triggers... When you are culture flipping others, you get the close borders and the alarming influence factors, fine. When your target planets flip and revolt against them, if you aren't putting more pressure on them at the moment, it clears those flags and suddenly things are fine again. You think that they'd be ticked about it instead of improving relations with you. These flags just don't work the way they are.