Rogue Agents in Multi-Agent Systems
- Inter-Agent Communication
A malicious or compromised agent operates outside its intended boundaries inside a multi-agent architecture, exploiting inter-agent trust to manipulate decisions, corrupt data, or execute unauthorized actions undetected.
What it is
Rogue agents in multi-agent systems emerge when a malicious or compromised agent operates outside the boundaries the architecture assumes for it, exploiting inter-agent trust, workflow dependencies, or shared resources to manipulate decisions, corrupt data, or run a denial-of-service attack from inside the system rather than against it. The threat is an adversarial exploitation of Excessive Agency specific to multi-agent settings: because the rogue agent is a legitimate participant with a legitimate identity, its actions look like ordinary agent behavior to any check built around 'is this a valid agent,' and it can remain embedded in a workflow, unnoticed, well past the moment it was compromised or introduced.
A rogue agent can arrive two ways: an adversary can stand one up directly and insert it into the system, or an agent that was legitimate at deployment can be compromised in place and turned rogue during operation — including through an infectious backdoor, where a single compromised agent embeds malicious logic in its own reasoning chain and other agents silently inherit it simply by consuming its output during ordinary inter-agent coordination, so the compromise spreads without any single message looking like an attack.
Kinds
- Externally introduced rogue agent
- An adversary stands up a malicious agent from outside and inserts it into the system, where it exploits inter-agent trust to act as if it were a legitimate participant.
- Infectious backdoor
- A single agent already inside the system is compromised, and other agents that consume its output during ordinary coordination silently inherit its malicious logic, letting the compromise spread without a distinct introduction event.
Attack scenarios
A compromised specialist agent impersonates a financial-approval role and injects a fraudulent transaction that downstream agents process because inter-agent trust was never independently verified.
Malicious workflow injection
A rogue agent impersonates a financial-approval role, exploiting inter-agent trust to inject a fraudulent transaction while bypassing validation controls.
Orchestration hijacking
A rogue agent routes a fraudulent transaction through multiple lower-privilege agents, leveraging fragmented approvals to bypass manual verification that a single higher-privilege check would have caught.
Coordinated agent flooding
Multiple rogue agents simultaneously generate excessive task requests, overwhelming computing resources and delaying the system's critical decision-making processes.
Infectious backdoor cascade
A single compromised agent in a financial multi-agent system embeds an infectious backdoor in its reasoning chain; as other agents consume its outputs during coordination, the malicious logic silently propagates until transaction approvals across the network are systemically compromised.
Documented incidents
Public cases, each checked against its primary source. A research demonstration shows the attack works against real systems; it is not a reported breach.
Agent session smuggling over A2A
A malicious remote research agent used an ongoing A2A session to slip hidden instructions to a financial-assistant agent, which leaked data and executed an unauthorized stock trade. The protocol itself was not flawed; the attack abused the default trust agents place in the peers they collaborate with.
Palo Alto Networks Unit 42, “When AI Agents Go Rogue: Agent Session Smuggling Attack in A2A Systems”Prompt Infection between agents
In a simulated multi-agent system, one compromised agent passed a self-replicating injection to its peers through normal messages, spreading “much like a computer virus” until other agents acted on the attacker's instructions.
Lee & Tiwari, “Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems” (arXiv:2410.07283)Mitigations
- Constrain autonomy with policy
- Restrict agent autonomy using policy constraints and continuous behavioral monitoring; where cryptographic attestation for LLMs doesn't yet exist, controlled hosting environments substitute as the integrity boundary.
- Bound the blast radius
- Least Privilege Agent limits what any single agent, rogue or not, can reach, so a compromise stays contained rather than cascading across the system.
- Detect and isolate
- A Controller runs real-time detection to flag agents acting outside predefined policy, isolates a detected rogue agent along with its communication history and memory, and revokes its privileges pending review.
- Red-team and re-verify continuously
- Regular AI red teaming and input/output monitoring surface deviations before they cascade, and tracking rejoin attempts catches a previously disabled rogue agent trying to re-enter under a different identity.