Prompt Injection
- Input / Prompt
Crafted input overrides the model's instructions, directly or via poisoned content the agent reads (indirect injection).
What it is
Prompt injection is an unintended change in the model's behavior or output driven by crafted input. It differs from jailbreaking in scope: jailbreaking specifically targets a model's safety guardrails, while prompt injection is the broader class of unintended behavior change, whether or not safety is the target. Retrieval-augmented generation and fine-tuning improve relevance but do not close the gap, because the vulnerability lives in how the model interprets text, not in what it was trained on. Agentic and multimodal systems widen the exposure considerably: an agent that reads tool output, web content, or images treats each as a potential instruction channel, and defenses built only to filter text miss attacks embedded in other modalities.
Kinds
- Direct injection
- The attacker's own prompt tries to override the system's instructions directly — intentionally, to bypass guardrails, or by accident, when an ordinary input triggers unexpected behavior.
- Indirect
- The model ingests instructions embedded in content it retrieves — a webpage, a document, a tool result — and follows them as if the user had typed them, without the user ever seeing the injected text.
- Multimodal
- Instructions are hidden in a non-text channel — an image, audio, or benign-looking text paired with adversarial content — exploiting guardrails built to filter text alone.
Attack scenarios
A web-search tool's result contains a hidden instruction that a research agent follows as if it came from the user, silently changing the agent's next action.
Support bot bypass
An attacker submits crafted input to a customer-support chatbot that overrides its guidelines, coaxes it into querying private data, and drives it to send emails on the attacker's behalf.
Poisoned webpage summary
An LLM asked to summarize a webpage processes a hidden instruction embedded in the page, which causes it to insert an image whose URL silently exfiltrates the conversation.
Split payload in a resume
An attacker uploads a resume with a malicious instruction fragmented across several sections; individually innocuous, the fragments combine once the model reads the whole document and skew its evaluation.
Multimodal image injection
An instruction hidden inside an image, paired with unremarkable accompanying text, alters the model's behavior — a channel most text-only content filters never inspect.
Mitigations
- Constrain model behavior
- Give the model explicit role instructions in the system prompt, keep it bound to its defined task, and instruct it to refuse attempts to alter its own instructions — the same discipline behind Least Privilege Agent.
- Validate output formats
- Specify the expected output format, require the model to cite its sources or reasoning, and check every response against that format with deterministic code rather than trusting it by default.
- Segregate & tag external content
- Mark untrusted content — tool results, retrieved documents, web pages — as distinct from the user's own instructions, so the model and any downstream guardrail can weight it differently.
- Require human approval
- Gate high-risk operations behind a HITL Approval Gate so a successful injection still cannot act without a human sign-off.