Glossary
The vocabulary, defined plainly
Every term here is one somebody has had to explain in a meeting. Definitions lead with the plain answer rather than the marketing one - and where a word is commonly used loosely, the entry says so instead of quietly picking a side.
- AI gateway
A layer between your applications and the model providers. You integrate once; routing, fallback, spend limits, access rules and logging then apply to every request, whichever model answers it. The alternative is N integrations and N sets of rules, only some of which anyone remembers to update.
- BYOK
Bring your own key: the gateway calls providers with your API keys, so the provider bills you directly and negotiated rates and committed-use discounts keep working. What the gateway still supplies is everything your keys alone do not - one endpoint, routing, fallback, caps and one log.
- Context window
The maximum number of tokens a model can consider at once, prompt and answer together. Exceeding it is not a soft failure - the request is refused or silently truncated, depending on who sits between you and the model.
- Cost attribution
Knowing which app, customer or feature caused which part of the bill. Provider invoices cannot answer it, because the provider never knew your customers: the labels have to be attached at request time by whatever sits in front.
- Data residency
Where your data is processed and stored. Worth checking twice, because a platform can be hosted in one region while the models it calls run somewhere else. EU-hosted and processed in the EU are separate claims, and usually only one of them is being made.
- Embeddings
Numeric vectors representing the meaning of text, so similar meanings sit near each other. They are what retrieval and semantic caching are built on: you search the vectors, not the words.
- Fallback
Having a second answer ready before the first fails. Real fallback moves the request to a different model, usually from a different company, with the conversation intact. Retrying the same model inherits the same outage, which is why a claim to retry and a claim to fall back are not the same claim.
- Guardrails
Rules about what may be asked, what may be sent and which models may answer. The defining question is where they run: guardrails that scan logs afterwards produce incident reports, and guardrails that run in the request path produce prevented incidents.
- LLM routing
Choosing which model answers a request, per request, instead of hard-coding one. Useful because a one-line question and a full refactor are not the same job and should not cost the same. The question worth asking of any router is whether it tells you why it chose.
- MCP
The Model Context Protocol, a standard way for an AI assistant to call external tools and read external data. Connecting a gateway's MCP server lets the assistant writing your integration look up live models and prices instead of hardcoding a guess that drifts.
- Model alias
A name that resolves to a real model at request time - the strongest one a provider has, say - so your code needs no edit the day a better model ships. The trade is that you stop knowing exactly what answered, unless the response tells you.
- Observability
Being able to answer why one specific request behaved as it did: which model, why it was chosen, what it cost, what fell over. Distinct from monitoring, which answers whether things are up and fast in aggregate.
- PII redaction
Removing personal data from a prompt before it reaches a model. The test of an implementation is simple: does the provider receive the redacted text, or does it receive the original and the redaction happens in your logs afterwards?
- Prompt caching
A provider storing the long, unchanging front of your prompt so you are not billed full price to send it again. Agents resend system prompts and tool definitions constantly, so the saving is larger than it sounds. Read tokens are cheaper than fresh ones; writing the cache usually costs slightly more than not caching at all.
- Prompt injection
Text that tries to override the instructions a model was given, often arriving inside content your application fetched rather than from the user directly. Serious detectors decode evasions first, because attackers do not write the phrase in the clear: they base64 it, hex it, or scramble the letters.
- RAG
Retrieval-augmented generation: finding relevant documents and putting them in the prompt so the model answers from your material instead of its memory. In cost terms it is an embeddings workload and a generation workload sharing one bill, which is a good argument for putting both behind one key.
- Rate limit
A ceiling on requests per unit of time. Worth distinguishing from a spend cap: a rate limit says you are going too fast, a spend cap says you have gone far enough. They deserve different status codes and different client behaviour.
- Routing policy
A named, ordered set of models a request should try, for when you want control rather than delegation. Policies are how this workload must stay on EU-processed models becomes a rule instead of a convention.
- Semantic cache
Answering a question that means the same as an earlier one from the stored answer, rather than only an identical one. It compares embeddings and accepts a match above a similarity threshold. Deliberately unsuitable wherever close enough is not a property you want, such as structured output.
- Shadow AI
Employees using AI tools the organisation has not approved and cannot see. Usually a symptom rather than a cause: people route around the sanctioned path when it is slower or more restrictive than the alternative.
- Spend cap
A ceiling on cost rather than speed, enforced before a request runs. The useful ones exist at more than one level - account, key and session - because a runaway agent loop is a different failure from a month that quietly overran.
- Structured output
Making a model return JSON matching a schema you supplied, rather than prose you then have to parse. Support is uneven across providers, which is why a gateway translates one schema into whatever mechanism each provider actually implements.
- Time to first token
How long a streaming response takes to begin, as opposed to how long it takes to finish. It is the number a person waiting at a cursor actually experiences, and it is why total latency alone compares models poorly for interactive work.
- Token
The unit models read, write and bill in, roughly a short word or a fragment of one. Prices are quoted per million tokens, and input and output are priced separately, with output typically several times dearer.
- Tool calling
Letting a model ask your code to run a function and hand back the result. It is what turns a chat model into an agent, and the reason a gateway must preserve tool definitions and tool history across a fallback: a chain that drops them resets the conversation.
- Zero data retention
An arrangement where a provider does not keep prompts or completions at rest. Availability varies by provider and often by contract, which is why an honest catalogue records retention per provider with the date it was checked rather than as a blanket promise.
Most of these become one setting
Routing, fallback, caching, guardrails and spend caps are separate problems until they sit behind one endpoint, at which point they are configuration rather than architecture.