Routing by price, latency or quality - what the scorer actually does

Every request that arrives at Final Router has to be answered by exactly one model, and something has to choose it. Calling that something a router makes it sound clever. It is not clever, and it should not be - a routing decision you cannot reconstruct from the response is a routing decision you cannot trust with your production traffic.

So this post is the whole thing: what the scorer sees, what it does with it, and what it deliberately refuses to do.

The inputs

Four things go in. Three come from your account, one from the request.

• The models you have left switched on in the dashboard. A model turned off is one the router genuinely cannot reach - not one it prefers to avoid.
• The providers you can actually authenticate with: ours, plus any provider you have supplied your own key for.
• Your default routing strategy, which the request can override with a routing field.
• A specific model, if you named one. Naming a model pins the request to it and skips scoring entirely.

The intersection of the first two is the candidate set. If it is empty, you get a 503 that tells you which of the two causes it was - nothing switched on is a setting, no provider is a missing credential, and confusing the two costs an afternoon.

The four strategies

Each strategy is a single sort over the candidate set. That is the entire mechanism.

• cost - cheapest reachable model that clears a quality floor. The floor exists so that a request to summarise a legal document does not land on a model that cannot read one.
• latency - lowest observed median latency, measured from our own traffic rather than a vendor benchmark.
• quality - highest quality score, price ignored.
• balanced - the highest quality per cent spent. This is the account default and where most traffic belongs.

Balanced is the interesting one, because it is the only strategy that has to combine two units that do not naturally combine. We divide the quality score by the blended cost per million tokens and sort on the result. It is a crude ratio and we are not going to dress it up as anything else. Crude and inspectable beats sophisticated and opaque when the output is a bill.

What comes back

Every response carries a final_router object. It exists because the first question anyone asks about automatic routing is why did it pick that one, and the answer should not require a support ticket.

"final_router": {
  "strategy": "balanced",
  "reason": "Best quality-per-cent among reachable models",
  "used_own_key": false,
  "cost_cents": 0.41,
  "balance_cents": 4821.6,
  "fell_back_from": []
}

Clients that ignore unknown fields never see it. Clients that are debugging get the strategy, the reason, the cost of that specific request, the balance afterwards, and every model that was tried and failed before this one answered.

What the router refuses to do

The refusals matter more than the features. Three in particular.

1. It does not read your prompt to make a decision. Content-aware routing - classify the request, then pick a model for that class - is a real technique and we may build it one day. But it means an extra model call in front of every request, and a routing decision that changes when a user rephrases a sentence. Not by default, and never silently.
2. It does not silently substitute a model you asked for. An unrecognised model id used to be treated as auto, which meant a typo quietly got whatever the router picked, from a different vendor, billed as normal. Now it is a 400 that names the field.
3. It does not route around a provider because that provider is expensive for us. We charge on credit purchase, not on tokens, specifically so that this incentive does not exist. A gateway that marks up inference has a reason to route you badly, and you would have no way to tell.

Picking a strategy

In practice the split is boring, which is the point. Anything a human is waiting on with a cursor blinking goes on latency. High-volume classification, extraction and tagging goes on cost. Reasoning and code generation goes on quality. Everything else stays on balanced.

You do not have to pick globally. Set the account default to balanced, then send routing on the two or three endpoints where it genuinely matters. That is one field per call site, and it is the whole configuration story.