Structured output

json_schema on every model, not just the ones that support it

Most providers never implemented OpenAI's json_schema. The gateway translates it into the one thing every provider does honour - a forced tool call - and hands back plain JSON, parse-checked, streaming included. Your generateObject code works unchanged, against all seventeen models.

One schema contract on all seventeen models - including providers that never implemented json_schema natively
Parse-checked server-side: you receive valid JSON or a retryable 502 with nothing charged
generateObject, LangChain and plain OpenAI SDK calls work unchanged - no adapter code

Your schema is injected as one forced tool - the mechanism every provider honours - then unwrapped and parse-checked. Invalid JSON is a 502 and is not charged.

The request - unchanged

// The OpenAI shape you already know - on ANY model
{
  "model": "mistral/mistral-large-latest",
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "name": "ticket_triage",
      "schema": {
        "type": "object",
        "properties": {
          "severity": { "enum": ["low", "medium", "high"] },
          "summary":  { "type": "string" }
        },
        "required": ["severity", "summary"]
      }
    }
  },
  "messages": [{ "role": "user", "content": "..." }]
}

The response - plain JSON

// Plain JSON content, parse-checked before you see it
"choices": [{
  "message": {
    "content": "{\"severity\":\"high\",\"summary\":\"Checkout is down\"}"
  },
  "finish_reason": "stop"
}]

// If the model fails to produce valid JSON for the schema:
// 502 — and nothing is charged.

How the translation works

Your schema is injected as a single tool the model is forced to call - the one structured-output mechanism all seven providers implement. The gateway then unwraps the tool call back into ordinary message content, parse-checks it, and your client sees exactly what OpenAI-native json_schema would have returned. Streaming delivers the JSON as a single content delta with real usage in the final frame.

The honesty rules: a model that fails to produce valid JSON for your schema returns a 502 and is not charged - never a mangled payload. And because your own tools can't coexist with the forced tool, that combination is refused with a clear 400 instead of failing strangely downstream.

What that unlocks

Structured output stops being a model-selection constraint and becomes a request parameter.

One shape, seventeen models

Route structured extraction to the cheapest model, escalate to the strongest - the schema contract holds either way, so routing strategies and aliases work for structured traffic too.

"model": "auto" + json_schema: a routed, schema-locked answer

Guaranteed parseable, or free

The gateway parse-checks the JSON before returning it. Invalid output is a 502 with nothing charged - the failure mode is an error you can retry, never corrupt data you stored.

parse-checked server-side; 502 = uncharged

SDK defaults just work

Vercel AI SDK's generateObject, LangChain's structured output, or hand-rolled OpenAI SDK calls - anything that speaks response_format works without a provider-specific branch.

no adapter code, no per-provider JSON prompt tricks

Sharp edges are named

json_schema with your own tools is a 400 that says why. json_object is honoured natively where providers support it, and providers without a JSON mode are skipped for it.

named 400s beat mysterious provider errors

Through the gateway vs per-provider workarounds

	Final Router	Direct integration
json_schema coverage	Every model, all seven providers	Only providers that implemented it
Invalid JSON	502, uncharged, retryable	Parse errors in your code, tokens billed
Streaming	One content delta + real usage in the final frame	Provider-dependent, often unsupported
Model switching	Change the model string; the contract holds	Re-test every prompt trick per model
Cache correctness	Schema is part of the cache key - no cross-contamination	Your problem to remember

FAQ

Does this really work on models without native json_schema?

Yes - that is the point. Every provider behind the gateway implements forced tool calls, so the schema rides in as a tool definition and the response is unwrapped back to plain content. The capability flag json_schema: true in GET /v1/models is set for every chat model, and each was exercised live before the flag shipped.

Can I use my own tools and json_schema together?

No, and the refusal is explicit: the translation works by forcing a single tool, so caller tools cannot coexist with it in one request. You get a clear 400 rather than a request that silently ignores half its instructions. Split the call: tools first, then a structured summarisation.

What happens with streaming?

The structured JSON arrives as a single content delta, finish_reason stop, with real usage and cost in the final SSE frame - so streaming clients need no special handling beyond what they already do.

Is structured output cached?

The exact-match cache applies, and response_format is part of the cache key - a json_schema request can never collide with a plain request that has the same messages. The semantic cache deliberately skips structured requests: 'close enough' is not a property schemas have.

What does it cost?

Token prices are unchanged - the injected tool definition counts as normal input tokens at the model's list price, and a failed generation that returns 502 is not charged at all.

Pick the model for the job, keep the schema contract

Send your existing generateObject code at the base URL and point it at any of seventeen models.

Start free - 25 requests a day Read the quickstart