When to reach for it
- Agents act close to production.
- Tool and workflow boundaries must be verified.
- Regressions across versions must become visible.
Integration Tests for Agents.
Replayable end-to-end scenarios — fixtures, traces, evals.
Agents are tested across realistic scenarios, tools, memories, and control flows. Because output is non-deterministic, assertions check behavioural properties ('called the refund tool at most once', 'stayed within budget', 'refused the out-of-scope request') rather than exact strings, and at scale graduate into a simulation environment that replays thousands of synthetic scenarios.
In practiceA customer-service agent is tested nightly with 500 synthetic conversation scenarios; each run is judged against assertions such as 'called the refund tool at most once' and 'never disclosed PII', and failures block the release.
Without itWithout integration tests, behavioural regressions introduced by a prompt change or model upgrade go undetected until a production incident surfaces the broken pattern.
When it backfires
- Only an isolated prompt is being explored.
- Tests cannot be evaluated deterministically enough.
- The test environment endangers external systems.
The tradeoff
Higher operational security is gained against expensive test data, mocks, and evaluation logic.
What it actually does.
End-to-end runs are asserted against expectations.
Two ways this pattern will hurt you.
Tests pass by being too lenient
The test asserts only that the response is non-empty. The agent regresses to a generic 'I cannot help with that' and every test stays green.
Fix · Assert against typed state invariants and explicit content checks, not just liveness. Pair structural assertions with LLM-as-judge for semantic quality.
Flaky non-determinism gets muted instead of fixed
A behavioural assertion passes on most runs and fails on some. Under release pressure the team retries until green or deletes the test — and a real intermittent regression now ships silently.
Fix · Run each scenario N times and assert a pass rate, not a single pass. Seed and record fixtures so failures replay deterministically; quarantine, never delete, a flaky case.
Where Integration Tests for Agents is native.
LangSmitheval datasets + CINative
OpenAI Evalsscenario gradersNative
Microsoft Agent Frameworkevaluation harnessNative
Keep going