When to reach for it
- Quality criteria can be explicitly formulated.
- Iterative improvement is measurable.
- Creative outputs need rigorous checking.
Evaluator-Optimizer.
Generate. Judge. Improve. Repeat.
A generator produces a result, an evaluator scores it against specific criteria, and the generator optimizes it based on the feedback.
When it backfires
- The evaluator cannot provide reliable signals.
- Schema validation suffices.
- The budget does not allow for multiple runs.
The tradeoff
Better output quality is gained against additional evaluation complexity and latency.
A shape you can draw on a napkin.
An optimizer generates drafts; an external evaluator scores and feeds back until the threshold is met.
Two ways this pattern will hurt you.
Evaluator and optimizer share the same blind spot
Both models miss the same factual error. The loop converges to a polished but wrong answer.
Fix · Use a different model family for evaluator, or ground scoring in external checks (tests, regex, database queries).
Optimization metric drifts from user intent
The evaluator optimizes for BLEU score or length, not the user's actual quality criteria. The output looks good but misses the point.
Fix · Align the evaluator's rubric with user requirements. Calibrate with human ratings before deploying.
Where Evaluator-Optimizer is native.
Google ADKNative
LangGraphNative
AutoGen / AG2Native
Microsoft Agent FrameworkNative
Anthropic Cookbookdocumented as a recipeAdaptable