LLM observability
Know what your AI did, what it cost, and why
Observability here starts in the response itself: every completion carries which model answered, why it was chosen, what it cost and what fell over on the way. The dashboard aggregates the same truth - it never knows more than you were told.
- Cost in cents, the routing reason and the fallback trail arrive on the response itself - the primary record
- Analytics are totals computed in the database, split by app, session, tag and model - never samples
- Prompt text is stored only if you opt in: observability with consent, not surveillance by default
What you can see
Built for the three people who ask different questions: the engineer debugging a request, the owner watching the bill, and whoever answers when something breaks at 2am.
A request log that explains itself
Every request searchable, with a per-request timing waterfall - auth, guardrails, provider, billing - and filters that live in the URL so a view can be shared.
CSV export of exactly the filtered view, formula-injection guarded
Analytics that total, not sample
Cost, tokens, latency, failures and sessions over any window - aggregated in the database, so numbers are totals over everything, not the first thousand rows.
by model, provider, app title, session and tag
Cost on every response
final_router.cost_cents rides along with each completion, and streamed responses carry real usage in the final frame - no estimating from token counts.
cached reads itemised at the provider's discounted rate
Provider health, measured
Short probes hit every provider every fifteen minutes. The results feed routing, the alerting sweep, and a public status page anyone can check.
the /status page runs on the same data - no separate truth
Alerts that watch for you
Provider outages caught with zero traffic, graded minor or major by how many probes failed. Budget tiers at 50/75/90/100%. Pace projections before the overrun. One email per sweep, mirrored to a webhook.
alerts only when a model does not work; throttling is not an email
The pipeline watches itself
A separate check catches our own telemetry failing - because the dangerous outage is the one where the graphs look calm precisely because nothing is being recorded.
silence is treated as a symptom, not as good news
A digest that stays quiet
A daily owner email naming the models that failed most and the slowest by latency per output token - and nothing at all on a day with no traffic, because a mail that always arrives stops being read.
silent on quiet days, by design
Privacy is the default
We keep what a request cost and how it behaved - never what it said, unless you turn storage on. Then it is 30 days, only you can read it, and off deletes everything.
io logging: off by default, per-account switch
The response is the first dashboard
Most observability bolts a dashboard onto a black box. Ours starts one layer earlier: the final_router block on every response is the primary record - routing reason, fallback trail, cost, cache outcome. The dashboard is an aggregation of what your own client already received.
That ordering matters for trust: there is no number in our analytics that your logs cannot reproduce, and no silent behaviour the response didn't disclose at the time.
"final_router": {
"provider": "anthropic",
"cost_cents": 0.84,
"reason": "Chose claude-haiku-4-5 - lowest latency among healthy models.",
"fell_back_from": [],
"cache": { "hit": null, "saved_usd": 0 }
}
// The log keeps the same story: cost, latency waterfall,
// outcome, app title, session id — and the prompt text
// only if you switched storage on.Why the business cares
Observability is not a developer luxury - it is how AI spend becomes a managed cost instead of a surprise.
Costs stay attributed
Per-app titles, per-session ids and request tags mean the invoice decomposes into products, customers and features - not one blended number nobody owns.
Failures surface early
Grouped errors with a last-seen, probe-driven outage alerts and pace projections mean the bad week announces itself on day one, not on the invoice.
Savings get proven
The caching receipt is signed - writes show their cost next to the wins - so the savings number survives an audit instead of an applause round.
Observability with consent vs surveillance by default
| Final Router | Capture-everything tooling | |
|---|---|---|
| Prompt text | Stored only if you opt in; 30-day retention; off deletes all | Every prompt and response retained by default |
| Cost per request | On the response itself, in cents | Reconstructed later from token counts |
| Routing decisions | Reason stated per request | Not applicable - or not disclosed |
| Provider health | Public status page from real probes | Internal metrics, if any |
| Alert volume | One email per sweep, only when something is broken | A channel full of noisy warnings |
FAQ
Do you log our prompts and responses?
Not unless you switch it on. By default we keep what a request cost and how it behaved - model, tokens, latency, outcome - never the text. With storage on, text is kept 30 days, visible only to your account, and turning it off deletes everything stored immediately.
Can I see which app or customer drove the spend?
Yes. The X-Title header attributes requests to an app, X-Session-Id groups them into sessions with their own caps, and the user field plus tags label your own customers. Analytics breaks cost down along all of them.
What's the difference between monitoring and observability here?
Monitoring is the probes, the status page and the alerts - is it up, is it fast. Observability is the per-request story: which model, why, what it cost, what fell back, what the cache did. You get both, from the same data.
Do streamed responses lose usage data?
No. The final SSE frame always carries real usage - including cached-token counts - plus final_router.cost_cents. Streaming costs you nothing in accounting fidelity.
Can my team see the dashboards without touching billing?
Yes - invite them to the workspace. Members see traffic, analytics, usage and logs, and cannot touch keys, billing, guardrails or settings. Nothing they do spends your credit.
Bills that decompose, failures that announce themselves
Route one request through the gateway and read its final_router block - the observability starts there.
Related: the AI gateway, LLM caching, the live status page.