Automatic fallback
Provider outages your users never meet
Every provider has bad days - the question is whether your users find out. Every routing decision here carries a chain: when the chosen model fails, times out or throttles, the next one answers, and the response says so out loud.
- Chains can cross providers - an OpenAI outage falls through to Anthropic or Mistral, within whatever your policies allow
- Tool definitions and multi-turn tool history survive the failover, so agent conversations continue instead of resetting
- fell_back_from names every model that was tried - a silent substitution is the one thing fallback must never be
What is LLM fallback?
Fallback is the discipline of having a second answer ready before the first one fails. A gateway that merely retries the same model inherits the same outage; real fallback moves the request to a different model - often a different company - with the conversation intact.
Two things make ours trustworthy. It is probe-informed: short health checks hit every provider every fifteen minutes, so a struggling provider drops down chains before your traffic discovers the outage. And it is disclosed: the answering model is named in every response along with everything tried first, because reliability you can't audit is just a promise.
What the chain actually preserves
Failover is easy to claim and hard to do well - the details below are where most implementations quietly lose data.
Tools survive the walk
Tool definitions, tool_choice and the full multi-turn tool history are re-dialected for the next provider - an agent mid-task keeps its task, not just its last message.
tool_calls and tool results carried across providers
Probes see it first
Scheduled probes measure every provider every fifteen minutes. The same signal steers routing away from trouble and draws the public status page.
/status runs on the identical data - no separate truth
Named, never silent
fell_back_from lists every model that failed before the answer; policy.from_policy tells you when the safety net, not your policy, replied. Both are alertable fields.
the number worth monitoring: from_policy === false
Honest exhaustion
When every model in a chain is throttling, you get 503 overloaded_error with a Retry-After and nothing charged - not a fake success, not a vague 500.
all-throttled ≠ all-broken: the error says which
LLM fallback
Best practices - the order that works
- 01
Let chains cross providers
Same-provider fallback shares the same outage. The resilience is in the second company, not the second model - keep at least two providers in any chain that matters.
- 02
Pin only what must be pinned
A pinned model is a single point of failure you chose. Use auto, aliases or policies for everything that doesn't contractually require one exact model.
- 03
Alert on from_policy, not on errors
Fallback means your users see few errors - so watch the field that says the safety net answered. That is your early warning, days before an outage becomes a pattern.
- 04
Test the fallback path on purpose
Pin a chain, watch fell_back_from populate in the log, and confirm your client handles the answering model differing from the requested one. Reliability you have never exercised is a hypothesis.
- 05
Respect Retry-After when the chain is exhausted
A 503 overloaded_error means every option was throttling - retrying instantly joins the stampede. Back off for the stated seconds; nothing was charged.
Fallback here vs a retry loop in your code
| Final Router fallback | Client-side retries | |
|---|---|---|
| Provider outage | Next model answers, often another company's | Retrying the same dead endpoint |
| Tool conversations | History re-dialected and preserved | Each provider's format is your problem |
| Health awareness | Probe data steers chains before failures | Your first failed request is the probe |
| Disclosure | fell_back_from + reason on the response | Whatever your logging remembered |
| Cost of failure | Failed attempts are not charged | Depends on where the request died |
FAQ
Does fallback change what I'm charged?
You pay for the model that actually answered, at its list price - failed attempts along the chain are not charged, and a fully exhausted chain returns 503 with nothing charged at all.
Can fallback substitute a model I didn't allow?
No. Chains are built only from models your key and account settings permit - every allowlist, switch and guardrail holds during failover. Fallback changes which permitted model answers, never the set of permitted models.
What about a pinned model - does it still fall back?
Pinning narrows the chain per your configuration. If you need one exact model with no substitution, that is respected - and when it fails, you get its honest error rather than a silent stand-in.
How fast is the failover?
The chain walks immediately on failure - no separate retry timers - and probe data means genuinely unhealthy providers are usually demoted before your request ever tries them. The common case is that the first model in the chain simply already avoids the outage.
How do I know how often I'm falling back?
Three ways: fell_back_from on each response, the fallback-rate stat on the usage dashboard, and policy.from_policy in the log for the specific requests the safety net saved. It is a measured number, not a vibe.
The outage becomes a line in a log
Route through the gateway and the next provider incident is something you read about in fell_back_from - not in your support queue.
Related: smart LLM routing, LLM observability, the live status page, BYOK.