LLM caching
Stop paying twice for the same context
Agents resend the same system prompts, tool definitions and documents on every call. Final Router caches at three levels - the provider's, ours, and a semantic layer - and puts a signed savings number on every response, so the caching is a receipt, not a promise.
- Identical repeats are answered by the gateway - no provider call, no generation time, no token cost
- Near-duplicates match in the semantic layer above 95% similarity, with the score in the response
- Provider prompt caching is managed for you and billed through at the provider's discounted cached rate
Three layers, each with a different job
One cache can't cover repeated prefixes, identical requests and near-duplicate questions at once. Three can.
Provider prompt caching
Anthropic breakpoints are placed by the gateway; OpenAI, DeepSeek, Google, Kimi and Grok cache automatically. The provider's discount is billed through at the provider's own cached rate.
usage.prompt_tokens_details.cached_tokens on every hit
Exact-match response cache
An identical request within the hour is answered from the gateway with no provider call at all - no generation, no wait, no token cost.
keyed on a one-way hash; we store the answer, never the question
Semantic cache
Requests that mean the same thing - "what is our refund policy" vs "what's your refund policy?" - match above 95% similarity, with the score in the response.
cache.hit: "semantic", similarity: 0.97
The signed receipt
Every routed request reports what caching saved in dollars. Writes report negative, because embedding a request costs money and honest arithmetic includes it.
cache.saved_usd - negative on writes, by design
The rules that keep it honest
Caching that surprises people is worse than no caching. Ours follows four stated rules.
Temperature above zero is never cached
Asking for temperature is asking for variety; handing back the same answer twice would quietly take that away. Send cache: "exact" to override per request, or "off" to skip.
We store the answer, never the question
The lookup key is a one-way hash of the request, so two identical requests are recognised without the gateway holding either of them.
Don't send cache_control - we handle it
The gateway places Anthropic cache breakpoints itself and manages every provider's dialect, so there is no per-provider caching logic for your team to build or maintain.
Flush is one call
POST /v1/cache/flush drops every cached answer on the account - the deploy step for when the right answer just changed. It's free.
What a cached response looks like
The receipt is the feature. Cached reads appear in usage.prompt_tokens_details.cached_tokens - deliberately not folded into prompt_tokens, so your budget arithmetic needs no adjustment. Gateway-level hits carry the layer that answered and the dollars saved.
The analytics page totals it up: what caching saved this month, by layer, next to what the cache writes cost. Both numbers, because a savings dashboard that only shows wins is marketing, not accounting.
"usage": {
"prompt_tokens": 412,
"prompt_tokens_details": { "cached_tokens": 384 }
},
"final_router": {
"cache": { "hit": "semantic", "similarity": 0.97, "saved_usd": 0.0112 }
}
// saved_usd is SIGNED: a cache write shows as negative,
// because a savings figure that hides its own costs
// is one nobody should believe.With the gateway vs per-provider cache plumbing
| Final Router caching | Hand-rolled per provider | |
|---|---|---|
| Provider dialects | One integration; breakpoints and quirks handled centrally | Separate cache logic per provider, re-learned per model |
| Identical repeats | Answered by the gateway - zero provider cost | Full price unless you built a response cache |
| Near-duplicates | Semantic layer above 95% similarity | Reprocessed at full price |
| Proof it works | Signed saved_usd per request + monthly totals | Assumed, until someone audits the invoices |
| Discounted cached reads | Billed at the provider's cached rate, itemised | Available, if configured correctly everywhere |
FAQ
Does caching work across multiple providers?
Yes. The exact and semantic layers sit in the gateway, in front of every model, so they work even for providers with no cache of their own. Where a provider does offer prompt caching, the gateway manages its dialect and bills the discount through.
Does it replace provider prompt caching?
It complements it. Exact repeats never reach the provider; everything else is structured so the provider's own prefix cache fires - Anthropic breakpoints placed automatically, automatic caches left to do their work - and the cached_tokens receipt shows it happened.
Do I need to change my application?
No. Requests keep flowing through the same endpoint. The one rule: don't send cache_control yourself - the gateway owns cache placement, and sending your own returns a clear 400 instead of a silent conflict.
Is cached data secure?
The exact cache stores the response, keyed by a one-way hash of the request - the question itself is never stored. Caches are per-account, entries expire after an hour, turning the feature off deletes everything stored, and flush is a single free call.
What does the semantic cache cost?
It requires $25 of lifetime credit, because it is the one layer that costs us on every attempt - each request must be embedded whether or not anything matches. The exact cache and provider passthrough are free from day one.
Turn repeated context into a line item
Switch the cache on, keep your workflow, and watch saved_usd accumulate on requests you were paying full price for.
Related: the AI gateway, smart LLM routing, LLM observability.