Cascading Hallucination Attacks
- Inter-Agent Communication
- Output / Actuation
An agent's tendency to generate plausible-but-false content is exploited so the fabrication propagates and amplifies through memory, self-reflection, or inter-agent communication rather than staying contained to one response.
What it is
Cascading hallucination attacks exploit an agent's tendency to generate plausible-but-false content, engineering conditions so the fabrication propagates and amplifies rather than staying contained to a single response. The mechanism differs by scope: in a single agent, self-reinforcement through reflection, self-critique, or memory recall lets a hallucination compound across the agent's own repeated interactions with itself; in a multi-agent system, inter-agent communication loops let one agent's fabrication become a peer's trusted input, so the error propagates across the workflow rather than being independently re-checked at each hop. The threat extends Misinformation (LLM09) into a specifically agentic failure mode: a downstream agent has no way to distinguish a verified fact from an upstream hallucination unless the pipeline explicitly checks for it, so a single misclassification early in a pipeline can drive several later, seemingly independent decisions that are actually all downstream of the same original error.
Kinds
- Single-agent self-reinforcement
- Reflection, self-critique, or memory recall lets an agent's own earlier hallucination compound across further interactions with itself, entrenching the fabrication rather than correcting it.
- Multi-agent propagation
- Inter-agent communication loops let one agent's fabrication become a peer's trusted input, so the error propagates and amplifies across the workflow instead of being independently re-verified at each hop.
Attack scenarios
One agent misclassifies a financial-transaction anomaly as legitimate, and two downstream agents in the pipeline act on that classification without independently re-verifying it, propagating the wrong decision across the workflow.
Sales misinformation cascade
An attacker subtly injects false product details into a sales agent's responses; the fabrication accumulates in long-term memory and logs, so each later interaction compounds on the last and the misinformation worsens over time.
API hallucination and leakage
Hallucinated API endpoints introduced into an agent's context trick it into generating fictitious calls, leading to accidental data leaks and integrity compromise as the fabricated interface is treated as real.
Healthcare guideline drift
A false treatment guideline implanted in a medical agent's responses progressively builds on its own earlier hallucinations, producing dangerously flawed recommendations that compound rather than reset with each new query.
Cross-agent security desensitization
An agent hallucinates an incorrect security threshold and tells other connected systems that failed access attempts are low-risk, so an entire network of downstream agents adopts and acts on the same fabricated policy.
Mitigations
- Score outputs before they're trusted
- LLM-as-Judge checks generated claims against a rubric, and Statistical Guardrails flag content that drifts from retrieved or verified sources before it's passed downstream.
- Ground responses in verified retrieval
- Require the model to cite what it retrieved rather than reasoning from memory alone, so a claim has a checkable source instead of an unverifiable prior output.
- Monitor cross-agent consistency
- A Controller watches for agents making contradictory decisions on similar cases or repeating an upstream claim without independent re-verification, catching propagation before it compounds further.
- Re-verify at every hand-off, not just once
- Treat an upstream agent's output as untrusted input at each new hop — the same discipline the Cascading Security Vulnerabilities anti-pattern names for inter-agent messages generally.