Operational category
Observability
How you see what happened.
LLM-as-Judge
An LLM grades another LLM.
A model evaluates outputs against criteria, rubrics, or comparative examples — scoring one output at a time (pointwise) or picking the better of two (pairwise, more robust to score drift). Structuring the judgment as chain-of-thought reasoning before a form-filled score (the G-Eval paradigm) improves alignment with human ratings.
Trade-off Scalable evaluation is gained against uncertainty and calibration needs.
Distributed Tracing
Spans across the whole agent loop.
Agent runs, tool calls, and subprocesses are made visible as connected end-to-end traces. The cross-vendor standard is the OpenTelemetry GenAI semantic conventions (gen_ai.* span attributes), so a run instrumented once is portable across observability backends; a common sampling policy keeps all error traces and only a small fraction of successful ones.
Trade-off Deep diagnostic capability is achieved against telemetry and privacy management effort. Indispensable in production.
Token / Cost Tracking
Tokens and dollars, per run, per agent, per node.
Token consumption, model costs, and tool-related expenses are measured per run, agent, or workflow. The measurement is only useful if something acts on it: a kill switch (a hard budget ceiling that terminates a runaway run before a retry storm multiplies the bill) and caching of intermediate results turn the metric into a control.
Trade-off Better budget control is gained against additional measurement and aggregation effort.
Integration Tests for Agents
Replayable end-to-end scenarios — fixtures, traces, evals.
Agents are tested across realistic scenarios, tools, memories, and control flows. Because output is non-deterministic, assertions check behavioural properties ('called the refund tool at most once', 'stayed within budget', 'refused the out-of-scope request') rather than exact strings, and at scale graduate into a simulation environment that replays thousands of synthetic scenarios.
Trade-off Higher operational security is gained against expensive test data, mocks, and evaluation logic.
Where to next