Memory Poisoning
- Memory / State
Attackers exploit an agent's reliance on short- or long-term memory to inject malicious or false data, corrupting future decisions, bypassing security checks, or escalating privilege via memory recall.
What it is
Memory poisoning exploits an agent's reliance on stored context to corrupt what it treats as settled fact. Short-term (in-context) attacks exploit the limited context window to make an agent repeat a sensitive operation or load manipulated data within a single session; long-term attacks inject false information that survives across sessions, corrupting the knowledge base the agent later recalls as fact. An attacker reaches this surface either through a direct prompt injection into an agent's own isolated memory, or by exploiting a memory store several agents share. The threat extends Data and Model Poisoning (LLM04) from a static, training-time surface into a live, persistent one: the corrupted content wasn't baked in during training, it was written by an attacker during ordinary operation, and it stays wrong until something notices. Where memory is embedding-backed, Vector and Embedding Weaknesses (LLM08) compounds the risk further, since adversarial content can also skew what similarity search treats as relevant.
Kinds
- Short-term / context poisoning
- Exploits the agent's limited context window to force it into repeating a sensitive operation or loading manipulated data within a single session.
- Long-term / persistent poisoning
- False information injected once survives across sessions, corrupting the knowledge base an agent later recalls as settled fact.
- Shared-memory poisoning
- A memory store several agents or users read from is corrupted a single time, and every later consumer of it inherits the same false belief.
Attack scenarios
A shared-memory travel-booking agent has a false 'chartered flights are free' pricing rule repeatedly reinforced by an attacker, until it authorises unauthorized bookings without payment validation.
Context window exploitation
An attacker fragments a privilege-escalation attempt across many sessions so no single interaction looks suspicious, exploiting the agent's limited context to never recognize the pattern until access is already granted.
Security system desensitization
An attacker gradually retrains a security agent's memory to reclassify malicious activity as normal, one subtly mislabeled incident at a time, until real intrusions pass unflagged.
Shared refund-policy corruption
An attacker plants an incorrect refund rule in a customer-service memory store several agent instances read from, so every agent that consults it approves refunds it shouldn't, compounding the financial loss.
Persistent copilot exfiltration channel
A single indirect prompt injection delivered through an enterprise copilot's email inbox poisons its memory into a standing exfiltration channel that keeps leaking data on every later session, long after the original email is gone.
Mitigations
- Validate before it's stored
- Screen every candidate memory write for anomalies before it's accepted, restrict what may persist to trusted sources, and require source attribution so a later audit can trace where a belief came from.
- Scope and isolate access
- Segment memory by session and bind each agent's read access to only what its current task needs, per Least Privilege Agent, so a poisoned entry in one partition can't silently answer another agent's query.
- Detect drift and roll back
- Statistical Guardrails flag anomalous memory-modification patterns, and periodic memory snapshots make a poisoned entry forensically reconstructible and reversible through the Audit Trail.
- Verify before it commits long-term
- Require independent or multi-agent validation — and, where feasible, a probabilistic check against trusted sources — before a memory update is allowed to persist across sessions.