Data and Model Poisoning
- Memory / State
- Tools & External Data
Training, fine-tuning, or embedding data is corrupted — deliberately or via a compromised source — so the model's outputs or a retrieval index become unreliable in a way an attacker controls.
What it is
Data poisoning corrupts a model at the data layer — pre-training, fine-tuning, or embedding data manipulated to introduce a bias, a vulnerability, or an outright backdoor. Because tampering with training data changes what the model learns to predict, this is an integrity attack, and it is hardest to catch when it targets a backdoor: the model's behavior stays normal until a specific trigger fires, in effect creating a sleeper agent that ordinary testing never encounters. A multi-agent system raises the stakes because poisoned data rarely stays contained to one consumer — a shared retrieval index or knowledge base is, by construction, read by every agent that queries it, so a single falsified document propagates the same wrong answer across the whole system rather than staying an isolated response.
Kinds
- Training-data poisoning
- An attacker introduces harmful or biased examples directly into pre-training or fine-tuning data, exploiting how training pipelines ingest and weight data over time.
- Retrieval / embedding poisoning
- A falsified document planted in a shared knowledge base or vector index is retrieved and treated as ground truth by every agent that queries it, without the falsification ever touching the model's own weights.
- Backdoor insertion
- A trigger phrase or pattern is trained into the model so its behavior stays normal until that trigger appears, at which point it executes the attacker's intended deviation — hard to detect because ordinary evaluation never encounters the trigger.
Attack scenarios
An attacker seeds a knowledge base shared by several agents with a subtly falsified policy document, so every agent that retrieves from it inherits the same wrong answer.
Biased outputs via manipulated training data
An attacker biases a model's outputs by manipulating its training data, spreading misinformation as if it were an ordinary answer.
Falsified shared documents
A malicious actor seeds a knowledge base shared by several agents with subtly falsified documents; every agent that retrieves from it inherits and repeats the same wrong answer.
Toxic data without filtering
Unfiltered toxic training data propagates harmful or biased content into the model's outputs, with no single obviously malicious input to flag.
Backdoor trigger insertion
An attacker poisons training data to insert a backdoor trigger, later exploitable for authentication bypass, data exfiltration, or hidden command execution once the trigger appears in a live prompt.
Mitigations
- Track provenance and verify data
- Track data origin and every transformation with a bill-of-materials approach, and validate legitimacy at every stage of model or index development rather than trusting an upstream source by default.
- Partition and permission-scope memory
- Partition retrieval stores per Semantic / Vector / Graph Memory so a poisoned entry in one partition can't silently answer queries from an unrelated agent or tenant.
- Validate ingested content
- Screen everything entering a shared index or training set with the Integrator before it's accepted, catching manipulated or unverified content at the boundary rather than after it has already propagated.
- Monitor for drift
- Watch training loss and live output for the anomalous patterns Statistical Guardrails are built to catch — poisoning that changes behavior gradually is invisible to a one-time spot check.