Try to break it first
Threats, controls, and governance are a claim that a system is safe. Red teaming is the discipline that tries to falsify the claim before an adversary does — the offensive lens over the same threat surface.
Structured, adversarial testing of a GenAI system, extended for autonomous agents. It is the empirical evidence the VALID control function demands. The primary source is the OWASP GenAI Red Teaming Guide (v1.0, 2025), extended for agents by the CSA + OWASP Agentic AI Red Teaming Guide (2025).
What red teaming is — and what it is not
GenAI red teaming combines human expertise with automation to uncover safety (of users), security (of the operator), trust (by users and partners), and performance gaps across the whole stack — not just the model. It keeps the classic threat-model → recon → exploit → report arc, but adds four properties ordinary penetration testing does not handle:
- Wider scope of concern
Socio-technical risks — bias, harmful content, over-reliance — not only technical compromise.
- Data complexity
Curating and generating large, often multimodal adversarial datasets.
- Stochastic evaluation
A non-deterministic target means findings are statistical, over many trials — not a single pass/fail.
- Threshold-based criteria
The focus shifts from a one-time breach to statistical thresholds and continuous monitoring.
An eval suite is an input to red teaming, never a substitute: passing a battery of automated tests does not make a system secure, and failing one does not make it insecure. Benchmarks measure aggregate capability against a fixed set; red teaming hunts the adversarial edge and the emergent, multi-step failure a benchmark never scripts.
The four pillars of an engagement
The attack surface is structured into four pillars, moving outward from the model to the humans and agents that use it. Select one to read what it probes.
Intrinsic model weaknesses — toxicity, bias, alignment failures — plus model provenance, model-malware injection, and training-data-pipeline poisoning (the model development lifecycle).
Red-teaming an agent: twelve adversarial categories
When the target is an autonomous, tool-using, multi-agent system, an adversary can chain attacks across services and turns, manipulate the agent's decision-making, and bypass access control through agent interactions rather than against the model directly. The CSA + OWASP guide catalogues twelve test categories — each the offensive counterpart of a threat this reference already documents.
- 01
Agent Authorization & Control Hijacking
Seizing an agent's authority or control path to act with its privileges.
- 02
Checker-Out-of-the-Loop
Defeating or fatiguing the human/automated approver so oversight lapses.
- 03
Agent Critical System Interaction
Driving the agent to invoke destructive or high-impact tools and actions.
AttacksT2Tool Misuse - 04
Goal & Instruction Manipulation
Rewriting the agent's objective or plan via injected content.
- 05
Agent Hallucination Exploitation
Weaponizing a confident but wrong output as an action trigger.
- 06
Agent Impact Chain & Blast Radius
Measuring how far a single compromise propagates across the system.
- 07
Agent Knowledge Base Poisoning
Corrupting the retrieval/knowledge store the agent grounds on.
AttacksT1Memory Poisoning - 08
Agent Memory & Context Manipulation
Planting persistent state that steers later, unrelated sessions.
AttacksT1Memory Poisoning - 09
Multi-Agent Exploitation
Spoofing, tampering, or replaying messages between collaborating agents.
- 10
Resource & Service Exhaustion
Exhausting compute, rate limits, or budget — up to denial of wallet.
AttacksT4Resource Overload - 11
Supply Chain & Dependency Attacks
Compromising a tool, plugin, MCP server, or package the agent trusts.
AttacksT17Supply Chain Compromise - 12
Agent Untraceability
Acting without leaving an attributable, reconstructable audit trail.
Continuous adversarial assurance
Red-teaming an agent is explicitly not a one-time event: the guide requires re-testing after every fix and periodic checks integrated into the AI lifecycle, because a non-deterministic system that composes behaviour at runtime drifts away from any point-in-time result. The 2026 OWASP landscape folds coordinated adversarial testing, defensive validation, and continuous feedback into a single lifecycle-wide loop.
Red teaming is how the AIUC-1 requirements tagged VALID are actually met — B001 (third-party testing of adversarial robustness), D002 (testing for hallucinations), D004 (testing of tool calls), C010/C011 (testing for harmful and out-of-scope outputs) — some on a fixed cadence, such as harmful-output evaluation at least every three months. It is increasingly a regulatory expectation too: the EU AI Act's systemic-risk red-team evaluations and DORA's threat-led penetration testing, extended to agent decision-making and tool-invocation patterns.
A finding is not the end of the loop — it is dispositioned into risk management, remediated, and re-tested, so that “all gates green” is continuously re-earned rather than certified once.
Security