Smart LLM routing
Send every request to the model that earns it
A one-line question and a full refactor are not the same job, but a single-model setup bills them the same. The router scores every model your key can reach on price, latency and quality - and unlike a black box, it writes the decision into every response.
- Four strategies - cost, latency, quality, balanced - scored over published prices, measured latency and live probe health
- Named policies pin exact chains when delegation is the wrong tool; aliases like provider/best never go stale
- Every decision is stated in final_router.reason - deterministic, explainable, auditable
What LLM routing means here
Routing is choosing which model answers a request - and being accountable for the choice. Ours is deterministic and explainable: the same request under the same conditions routes the same way, and final_router.reason says why in a sentence.
Four strategies
Cost picks the cheapest healthy model, latency the fastest, quality the strongest, balanced the best trade. Set an account default, override per request.
"routing": "cost" | "latency" | "quality" | "balanced"
Named policies
When you need control rather than delegation: name the exact models a request tries and in what order, or spread traffic across providers by weight.
"model": "policy/eu-only" - four presets built in
Health-aware
Short probes hit every provider every fifteen minutes. A struggling provider drops down the chain before your traffic finds out the hard way.
the same signal drives the public /status page
Aliases as standing orders
provider/best, /fastest and /cheapest resolve at request time against the live catalogue - the day a better model ships, you are already on it.
resolution stated in final_router.reason, never silent
How a request routes
- 1
Eligibility
Your key's allowed models, switched-on catalogue entries and guardrail rules define the pool. Nothing routes to a model you turned off.
- 2
Scoring
The chosen strategy ranks the pool on published prices, measured latency and quality scores - plus live probe health.
- 3
The chain
The winner leads a fallback chain. Provider fails or throttles? The next model answers, tools and history intact.
- 4
The receipt
final_router.reason states the decision; fell_back_from lists anything that was tried first. No silent substitutions, ever.
// Per request - overrides the account default
{ "model": "auto", "routing": "cost", "messages": [...] }
// Or a standing order that survives every model release
{ "model": "anthropic/best", "messages": [...] }
// Every response says what happened and why
"final_router": {
"reason": "Chose deepseek/deepseek-v4-flash - lowest cost among healthy models.",
"fell_back_from": []
}Deterministic beats clever
Some gateways route by classifying each prompt's content with another model. We deliberately don't: content-sniffing routing is unpredictable under load, hard to debug, and impossible to explain to a finance team. Scored routing over published metrics gives you the cost savings and an audit trail.
| Final Router routing | One default model | |
|---|---|---|
| Model selection | Best-fit per request, by strategy or policy | The same model for everything |
| Cost | Cheap requests route cheap; frontier models only when asked for | Frontier price on every request |
| A provider outage | Chain answers from the next healthy model | Errors until someone ships a fix |
| New model releases | Aliases resolve to them automatically | A redeploy per application |
| Explainability | Decision stated on every response | Nothing to explain - or audit |
Where routing pays for itself
Coding agents
Agents fire hundreds of mixed-complexity requests. Route the lookups cheap and keep the frontier models for the hard reasoning, per request.
session caps stop the runaway loop the same hour
Support and chat triage
Routine questions get fast, cheap answers; escalations get the strongest model. Latency drops where it matters most.
"routing": "latency" on the hot path
Compliance-bound traffic
The EU-only guardrail routes exclusively to models whose recorded data policy processes in the EU - everything else routes freely.
processing region recorded with source and date
The router shows its own working
A router that cannot prove it saved anything is a preference, not an optimisation.
Every request the router decided carries a measured saving: the difference between the dearest model that request could have used and the one that answered, priced at the same token counts from the same table the biller uses. It is arithmetic on what actually happened, not a modelled estimate.
Two things are deliberately excluded. A request where you pinned the model counts nothing - that was your choice, and crediting ourselves for it would be marketing arithmetic. And a chain with only one candidate counts nothing, because there was no alternative to be cheaper than. Analytics totals the rest into a Saved by routing figure beside what you spent.
// On the request the router chose:
requested: "auto"
chain: [gpt-5.1, claude-sonnet-5, claude-haiku-4-5]
answered: claude-haiku-4-5
// Priced at the identical token counts:
dearest in chain $0.0293
what it cost you $0.0084
───────
saved this request $0.0209
// Pinned a model instead? This request counts zero.FAQ
Does routing work across multiple providers at once?
Yes. The pool spans every model your key can reach across all seven providers, and a single fallback chain can cross provider boundaries - an Anthropic request can fall back to OpenAI if that is what your policy allows.
How does the router choose a model?
By scoring, not by guessing: published prices, measured median latency, quality scores and live probe health, ranked by the strategy you chose. The decision is deterministic and stated in final_router.reason on every response.
Can I pin a request to one exact model?
Always. A catalogue id like openai/gpt-5-mini routes to exactly that model, with fallback only as configured. Routing is delegation you opt into - auto, a strategy, an alias or a policy - never something forced on you.
Does switching models break my prompt caching?
Provider prompt caches are per-provider, so a pinned model keeps its cache warm. The gateway's own exact and semantic caches sit in front of every model, so repeated requests are answered before routing happens at all.
What does routing cost?
Nothing on top: tokens are billed at the answering model's list price with no markup, so a cost-routed request is genuinely cheaper, not cheaper minus a fee.
Stop paying frontier prices for routine work
Set "routing": "cost" and let the receipts on every response show what the router saved.
Related: the AI gateway, LLM caching, the EU gateway.