What is an LLM gateway?
An LLM gateway is a single API endpoint that sits between an application and multiple large language model providers. The application sends every request to the gateway, and the gateway routes it to OpenAI, Anthropic, Google, Azure or a self-hosted model, returning one consistent response format regardless of which provider answered.
What an LLM gateway does
Without a gateway, every provider integration lives in your application: a different SDK, a different authentication scheme, a different response shape, a different error taxonomy, and a different place to look when the bill arrives. Adding a second provider means writing that twice; adding a third means the branching logic starts to leak into business code.
A gateway moves all of it behind one endpoint. Your application makes one kind of call. Everything that differs between providers — credentials, request translation, token accounting, retries, fallback — becomes configuration in the gateway rather than conditionals in your code.
How a request moves through a gateway
- 01Your application calls one endpointA single base URL and a single API key, usually in the shape the OpenAI API defines, so existing SDK code works unchanged.
- 02The gateway authenticates and applies policyIt checks your key, enforces quotas or budgets, and applies any routing rules — which model, which provider, which region.
- 03It translates and forwards the requestThe gateway rewrites the request into the target provider's format and calls it with the credentials it holds for that provider.
- 04It normalises the response and records what happenedThe provider's response is translated back into one consistent shape, streamed if you asked for streaming, and the token counts and cost are recorded before the response reaches your code.
- 05On failure, it retries elsewhereIf the provider errors or times out, the gateway can retry against a fallback model rather than returning the failure to your application.
When you do not need one
One provider, one workload, no cost-attribution requirement and no uptime commitment beyond what that provider offers — call the provider directly. A gateway is a component you now operate or a vendor you now depend on, and neither is free. The honest trigger for adopting one is a second reason to route, not the possibility of one.
The second-cheapest option is worth naming too: open-source gateways such as LiteLLM and Portkey's gateway core cost nothing to license and run on infrastructure you already have. If you have a platform team, that is often the right answer, and our gateway comparison covers the trade-off in detail.
Common questions
A general API gateway routes HTTP traffic and handles concerns common to any API: authentication, rate limiting, request shaping. An LLM gateway does that and adds concerns specific to language models — translating between provider request formats, counting tokens rather than requests, tracking cost per call, streaming partial responses, and failing over to a different model when one provider degrades.
Not on day one. The gateway earns its place at the moment you have a second reason to route — a cheaper model for bulk work, a fallback when your primary provider has an outage, a customer who requires a specific region, or a cost figure someone wants broken down by team. If none of those apply yet, calling the provider directly is simpler and correct.
Yes — every proxy adds a network hop. How much depends on where the gateway runs relative to your application and to the provider. Caching can more than offset it, because a cache hit removes the provider round trip entirely. Measure it against your own traffic rather than trusting a published figure.
A router is one feature of a gateway. Routing decides which model handles a request, by rule or by cost or by availability. A gateway is the whole layer that request passes through, which also holds credentials, enforces quotas, records what was spent and normalises the response. Some products do only routing; those are routers.
Many can. The common pattern is to treat a local runtime such as Ollama or vLLM as one more provider behind the same endpoint, so an application can move a workload from a hosted frontier model to a local one by changing the model string. Whether a given gateway supports this is worth checking directly — it is not universal.