Misinformation
- Output / Actuation
The model produces false or misleading content that reads as confident and authoritative, which users or downstream systems trust without independent verification.
What it is
Misinformation is false or misleading model output that reads as confident and credible. Its main cause is hallucination — the model fills gaps in its training data with statistically plausible content rather than verified fact — but biased or incomplete training data contributes too. The risk is compounded by overreliance: when a user or downstream system trusts generated content without independent verification, a model's confident wrong answer becomes a real decision made on false premises, whether that decision is a legal filing, a medical suggestion, or a dependency pulled into production code. A multi-agent system compounds this further because one agent's fabrication routinely becomes another agent's trusted input: a downstream agent has no way to distinguish a verified fact from an upstream hallucination unless the pipeline explicitly checks for it, so misinformation propagates and compounds across a workflow exactly as Cascading Hallucination Attacks describes.
Kinds
- Factual inaccuracies
- The model states something false as if it were established fact, and a user or downstream process acts on it.
- Unsupported claims
- The model asserts something with no basis behind it, especially damaging in sensitive domains like health or legal advice where an unsupported claim can be mistaken for verified guidance.
- Misrepresented expertise
- The model conveys unwarranted confidence, or false uncertainty, about a topic — misleading the user about how settled the underlying answer actually is.
- Unsafe code or package suggestions
- The model recommends a nonexistent or insecure library, which becomes directly exploitable once a developer trusts the suggestion and integrates it.
Attack scenarios
A research agent fabricates a plausible-sounding citation, and a downstream summarisation agent repeats it as fact in the final report without an independent check.
Hallucinated-package attack
An attacker identifies library names that coding assistants commonly hallucinate, then publishes malicious packages under those names — any pipeline that trusts the model's suggested dependency without verification inherits the payload.
Unverified downstream repetition
A research agent fabricates a plausible-sounding citation, and a downstream summarisation agent repeats it as fact in the final report without an independent check.
Confident wrong answer in a high-stakes domain
A chatbot answering health or legal questions states an unsupported claim with a confident tone, and the process built around it acts without verification, causing real harm before anyone catches the error.
Mitigations
- Score outputs against a rubric
- Use LLM-as-Judge to check generated claims or citations against a defined rubric before the output is trusted downstream, rather than accepting a confident tone as a proxy for correctness.
- Catch topical drift statistically
- Statistical Guardrails flag responses whose content diverges from retrieved or verified sources, surfacing likely fabrication before it reaches a user.
- Ground responses in verified retrieval
- Use retrieval-augmented generation over a verified source and require the model to cite what it retrieved, rather than relying on unaided recall for factual claims.
- Cover the failure class with tests
- Integration Tests for Agents specifically probing for fabricated citations or facts, since this failure class does not reliably surface in ordinary functional testing.