When to reach for it
- Human evaluation must be scaled.
- Quality criteria can be linguistically formulated.
- Regressions in agent responses need to be detected.
LLM-as-Judge.
An LLM grades another LLM.
A model evaluates outputs against criteria, rubrics, or comparative examples — scoring one output at a time (pointwise) or picking the better of two (pairwise, more robust to score drift). Structuring the judgment as chain-of-thought reasoning before a form-filled score (the G-Eval paradigm) improves alignment with human ratings.
In practiceA customer-service platform runs a GPT-4o judge nightly over a sampled 5% of agent responses, flagging answers that fail a tone-and-accuracy rubric before they surface in the weekly quality report.
Without itWithout an automated judge, semantic regressions — answers that are factually wrong yet structurally valid — accumulate undetected, since schema validators and log monitors have no rubric for correctness or tone. See the anti-pattern →
When it backfires
- Objective tests are available.
- The evaluating model shares the same bias as the generator.
- High legal binding force is required.
The tradeoff
Scalable evaluation is gained against uncertainty and calibration needs.
What it actually does.
A model scores another model's output.
Three ways this pattern will hurt you.
Judge bias — the evaluator shares the generator's training data
The judge model was trained on the same corpus as the generator. It rewards answers that sound correct rather than are correct.
Fix · Use a different model family for evaluation, or ground scores in verifiable checks (test cases, regex, database queries).
Correlated failures — judge agrees with the same hallucination
Both generator and judge hallucinate the same false fact. The judge gives a high score to a wrong answer.
Fix · Add a human calibration set. When judge and human disagree, retrain the rubric or switch to a non-LLM verifier.
Position, verbosity, and self-preference bias skew the scores
In pairwise judging the model favours whichever answer comes first and rewards longer, more confident-sounding text; worse, it scores its own generations higher than humans do (self-preference correlated with self-recognition).
Fix · Average both orderings to cancel position bias, normalise for length, and judge with a different model family than the generator. Calibrate against a human-rated set.
Where LLM-as-Judge is native.
LangSmithLLM-as-judge evaluators over datasetsNative
OpenAI Evalsgraders + eval harnessNative
Microsoft Agent Frameworkbuilt-in evaluationNative