/pattern/operational/observability/

Operational category

Observability

How you see what happened.

LLM-as-Judge

An LLM grades another LLM.

A model evaluates outputs against criteria, rubrics, or comparative examples — scoring one output at a time (pointwise) or picking the better of two (pairwise, more robust to score drift). Structuring the judgment as chain-of-thought reasoning before a form-filled score (the G-Eval paradigm) improves alignment with human ratings.

Trade-off Scalable evaluation is gained against uncertainty and calibration needs.

Distributed Tracing

Spans across the whole agent loop.

Agent runs, tool calls, and subprocesses are made visible as connected end-to-end traces. The cross-vendor standard is the OpenTelemetry GenAI semantic conventions (gen_ai.* span attributes), so a run instrumented once is portable across observability backends; a common sampling policy keeps all error traces and only a small fraction of successful ones.

Trade-off Deep diagnostic capability is achieved against telemetry and privacy management effort. Indispensable in production.

Token / Cost Tracking

Tokens and dollars, per run, per agent, per node.

Token consumption, model costs, and tool-related expenses are measured per run, agent, or workflow. The measurement is only useful if something acts on it: a kill switch (a hard budget ceiling that terminates a runaway run before a retry storm multiplies the bill) and caching of intermediate results turn the metric into a control.

Trade-off Better budget control is gained against additional measurement and aggregation effort.

Integration Tests for Agents

Replayable end-to-end scenarios — fixtures, traces, evals.

Agents are tested across realistic scenarios, tools, memories, and control flows. Because output is non-deterministic, assertions check behavioural properties ('called the refund tool at most once', 'stayed within budget', 'refused the out-of-scope request') rather than exact strings, and at scale graduate into a simulation environment that replays thousands of synthetic scenarios.

Trade-off Higher operational security is gained against expensive test data, mocks, and evaluation logic.

Search

Search patterns, frameworks, and pages.