When to reach for it
- Agents act close to production.
- Tool and workflow boundaries must be verified.
- Regressions across versions must become visible.
Integration Tests for Agents.
Replayable end-to-end scenarios — fixtures, traces, evals.
Agents are tested across realistic scenarios, tools, memories, and control flows. Because the output is non-deterministic, the assertions check behavioral properties — "did it call the refund tool at most once?", "did it stay within budget?", "did it refuse the out-of-scope request?" — rather than exact output strings. At scale this graduates into a simulation environment: thousands of synthetic scenarios replayed against the agent to surface tail behaviors a handful of hand-written cases never reach.
In practiceA customer-service agent is tested nightly with 500 synthetic conversation scenarios; each run is judged against assertions such as 'called the refund tool at most once' and 'never disclosed PII', and failures block the release.
Without itWithout integration tests, behavioral regressions introduced by a prompt change or model upgrade go undetected until a production incident surfaces the broken pattern.
When it backfires
- Only an isolated prompt is being explored.
- Tests cannot be evaluated deterministically enough.
- The test environment endangers external systems.
The tradeoff
Higher operational security is achieved against expensive test data, mocks, simulation harnesses, and evaluation logic.
What it actually does.
End-to-end runs are asserted against expectations.
Two ways this pattern will hurt you.
Tests pass by being too lenient
The test asserts only that the response is non-empty. The agent regresses to a generic 'I cannot help with that' and every test stays green.
Fix · Assert against typed state invariants and explicit content checks, not just liveness. Pair structural assertions with LLM-as-judge for semantic quality.
Flaky non-determinism gets muted instead of fixed
A behavioral assertion passes on most runs and fails on some. Under release pressure the team retries until green or deletes the test — and a real intermittent regression now ships silently.
Fix · Run each scenario N times and assert a pass rate, not a single pass. Seed and record fixtures so failures replay deterministically; quarantine, never delete, a flaky case.
Where Integration Tests for Agents is native.
LangSmitheval datasets + CINative
OpenAI Evalsscenario gradersNative
Microsoft Agent Frameworkevaluation harnessNative
Keep going