Operational category
Observability
How you see what happened.
LLM-as-Judge
An LLM grades another LLM.
A model evaluates outputs against criteria, rubrics, or comparative examples — scoring one output at a time (pointwise) or picking the better of two (pairwise, more robust to score drift). Structuring the judgment as chain-of-thought reasoning before a form-filled score (the G-Eval paradigm) improves alignment with human ratings.
Trade-off Scalable evaluation is gained against uncertainty and calibration needs.
Distributed Tracing
Spans across the whole agent loop.
Agent runs, tool calls, and subprocesses are made visible as connected end-to-end traces. The cross-vendor standard is the OpenTelemetry GenAI semantic conventions (gen_ai.* span attributes), so a run instrumented once is portable across observability backends; a common sampling policy keeps all error traces and only a small fraction of successful ones.
Trade-off Deep diagnostic capability is achieved against telemetry and privacy management effort. Indispensable in production: vendors and practitioners consistently report large reductions in mean-time-to-diagnose once end-to-end traces are in place.
Token / Cost Tracking
Tokens and dollars, per run, per agent, per node.
Token consumption, model costs, and tool-related expenses are measured per run, agent, or workflow. The metric is only useful if something acts on it: a kill switch (a hard budget ceiling that terminates a runaway run) and caching of intermediate results turn the measurement into a control. Variable execution paths make cost forecasting hard, and a single edge case that triggers a retry chain can cost many times a normal run — so the ceiling is a safeguard, not an optimization.
Trade-off Better budget control is achieved against additional measurement and aggregation effort.
Integration Tests for Agents
Replayable end-to-end scenarios — fixtures, traces, evals.
Agents are tested across realistic scenarios, tools, memories, and control flows. Because the output is non-deterministic, the assertions check behavioral properties — "did it call the refund tool at most once?", "did it stay within budget?", "did it refuse the out-of-scope request?" — rather than exact output strings. At scale this graduates into a simulation environment: thousands of synthetic scenarios replayed against the agent to surface tail behaviors a handful of hand-written cases never reach.
Trade-off Higher operational security is achieved against expensive test data, mocks, simulation harnesses, and evaluation logic.
Where to next