When to reach for it
- Request volume is high with mixed difficulty.
- Cost is a first-order constraint.
- A cheaper model handles a meaningful share of traffic without quality loss.
Resource-Aware Optimization.
Route by difficulty. Pay for what the task needs.
A router scores each request's complexity and dispatches it to the cheapest model that still meets the quality bar, under explicit token, time, and cost budgets, using prompt caching and model cascades.
When it backfires
- Every task needs the top model.
- The complexity rubric is unstable — mis-route cost exceeds the savings.
- The latency of an extra routing hop is unacceptable.
The tradeoff
Large cost savings are gained against the effort of tuning and continuously validating the complexity rubric.
A shape you can draw on a napkin.
A router scores request complexity and dispatches to the cheapest model that meets the quality bar; a failed quality check cascades up to a stronger model.
Three ways this pattern will hurt you.
Routing on the model's own confidence
Asking the LLM 'how confident are you, 1–10?' to decide routing is hallucination-prone and unstable; the router itself becomes the weakest link.
Fix · Score complexity with a deterministic / statistical classifier (length, task type, historical difficulty) and validate routes against a quality signal rather than self-reported confidence.
Silent quality erosion on the cheap path
Once most traffic is routed to the cheap model, regressions there go unnoticed — nobody is watching the cheap path.
Fix · Sample and evaluate cheap-path outputs continuously (golden tests / LLM-as-judge) and fall back to the strong model when the quality bar is missed.
A rubric that rots as the model fleet changes
A complexity rubric tuned for one model pair silently misroutes once a new, cheaper model matches the old expensive one — or a price change inverts the cost math.
Fix · Pin the rubric to a fixed evaluation set and re-benchmark whenever the model fleet or pricing changes; treat the rubric as versioned configuration, not a constant.
Where Resource-Aware Optimization is native.
RouteLLMpreference-trained router across a cheap/strong model pairNative
OpenRoutergateway-level model routingAdaptable
Keep going