Every LLM provider goes down. Not often, and rarely for long, but reliably enough that any application depending on one of them has an availability ceiling it did not choose. Fallback is the obvious fix, and most implementations of it are worse than no fallback at all.
Not every failure is the same failure
The naive version catches an error and tries the next model. That turns four categories of problem into one behaviour, and three of those categories get the wrong one.
- A 429 from the provider means try again shortly, or try elsewhere now. Falling back is right, and so is remembering it for the next thirty seconds.
- A 500 or a timeout means this provider is unwell. Fall back, and put the provider in a short cooldown so the next hundred requests do not each pay the timeout to learn the same thing.
- A 401 means the credential is wrong. Falling back hides a configuration error behind a working response, and you will find out weeks later when the bill from the other provider arrives.
- A content-policy refusal means the model declined. Trying four more models until one complies is not resilience - it is circumventing a safety system, and every provider's terms prohibit it.
So the chain distinguishes them. Rate limits and transient errors advance to the next model. Auth failures surface immediately as an error you can act on. Policy refusals return the refusal.
Cooldowns beat retries
The most useful piece of reliability work we did was not the retry logic. It was remembering.
When a provider fails in a way that suggests it will keep failing, it goes into a short cooldown and drops out of the candidate set. Requests route around it without paying the timeout first. When the cooldown expires it is quietly let back in, and if it fails again the window doubles. This is a circuit breaker, which is a solved problem - the only insight is that a router is the right place to put one, because it is the only component that sees every request.
Streaming changes the rules
Fallback works cleanly right up until the first token is on the wire. After that there is nothing to fall back to: the client has already rendered half a sentence, and starting a different model would either duplicate it or contradict it.
So streaming fallback applies before the first token and not after. If a provider dies mid-stream, you get a truncated response and a finish reason that says so. That is worse than a clean retry, and it is honest, which on balance is what you want from infrastructure.
Tell the caller what happened
A fallback that is invisible is a bug generator. Your p99 latency doubles, your cost per request moves, the answers change character - and every dashboard says success.
"fell_back_from": [
{ "model": "gpt-4.1", "status": 429, "message": "Rate limit reached" },
{ "model": "claude-sonnet-4.5", "status": 503, "message": "Overloaded" }
]Every attempt that failed is in the response, with the status and message the provider returned. The model field tells you who actually answered. Log both and a change in routing behaviour shows up in your own metrics on the day it happens, not in the invoice at the end of the month.
What we count as an outage
A request served by the third model in your chain is a success. The provider had an outage; you did not. That is the whole point of the chain, and counting it as a failure would mean measuring somebody else's uptime and calling it ours.
What we do count is the gateway failing to route at all - no candidate reachable, or the request never making it out of our infrastructure. That number is ours to own, and it is the one we publish.