Model routing means choosing a model per request instead of using one model for everything. The goal is the cheapest option that still meets your quality bar. Start with static routing by task type, add escalation on failure, and only build a learned router once you have evaluation data to train and judge it.
Three approaches compared
Static routing by task. A code path decides the model. Classification and extraction go to a small model. Planning and hard debugging go to a larger one. It is easy to reason about, easy to test, and needs no extra model call.
Cascade with escalation. Try a small model first. If a check fails, retry on a larger model. The check can be a schema validation, a test run, a confidence field, or a second model acting as judge. The cost is extra latency on escalated requests and a retry bill for the failed first attempt.
Learned or classifier routing. A small model or a trained classifier reads the request and picks a target. It can adapt to varied input, but it adds a component to evaluate, monitor, and keep honest as models change.
Start with the cheapest control
Before routing across models, check whether one model with a lower effort setting does the job. The provider documentation describes effort as a way to trade thoroughness for token efficiency with a single model, and suggests lower levels for simple, high volume work such as subagents. That can remove the need for a second model.
What to measure
Routing only helps if you can prove quality held. Build an evaluation set from real requests and record, per route:
- Quality score or pass rate against your own rubric.
- Total tokens, split into input, output, and reasoning.
- Latency.
- Escalation rate, if you use a cascade.
Compare cost per successful task, not cost per call. A cheap model that fails often and triggers retries can cost more than a stronger model that succeeds the first time.
Count tokens per model
Pre-flight token counting can inform routing, for example by sending very long inputs to a model with a larger context window. The token counting docs also warn that newer model generations tokenize differently, with about 30 percent more tokens for the same text in one comparison. Count against each candidate model rather than reusing one count.
Watch the cache
Switching models mid conversation means a cold cache on the new model, since caches are tied to the prefix and model configuration. Route per conversation or per task when you can, not per turn.
Guardrails for routers
- Log which route handled every request.
- Alert on a sudden shift in the share going to the expensive tier.
- Keep a manual override for incidents.
- Re-run the evaluation whenever you change a model or a prompt.
Quick check
Is a cascade always cheaper than always using the large model?
No. If the small model fails often, you pay for both attempts. Compare cost per successful task on your own data.
Do I need a learned router to start?
No. Static routing by task type captures most of the value with far less to maintain.
Sources
Last reviewed: 19 October 2026.
