Field notes: running an AI gateway in front of every model call
A gateway is a single door that every model request walks through. After six months with one in production, here is what it actually bought us.
An AI gateway is a proxy that sits between your applications and the model providers they call. Every request goes through it, and in exchange it gives you one place to handle keys, routing, retries, budgets and logging. This piece is about what that looked like for a mid-sized engineering team that put one in front of everything, and which of the promised benefits materialised.
Why we did it
We had four services calling three providers with keys scattered through environment variables. A pricing change at one provider took a week to fully roll out because each team had hard-coded model names. When one provider had an outage, three of the four services simply failed. None of this was exotic; it was the natural state of a team that had adopted models quickly.
What the gateway gave us
One interface. Every service now speaks a single request format and names a model by an alias. Swapping the model behind an alias is a configuration change, not a deploy. This alone paid for the migration.
Fallbacks and retries. When a provider returns an error or times out, the gateway retries and then fails over to a second provider. Outages became latency blips rather than incidents.
Budgets by team. Each service authenticates with its own virtual key, and each key has a monthly spend limit and a rate limit. Finance can see who spent what without asking engineering.
A single log. Every request, its latency, token counts and cost land in one place. This turned out to be the feature we used most: it is where we found the two endpoints responsible for most of our spend.
The gateway did not make the models better. It made our use of them legible.
What it cost
Latency overhead was small: single-digit milliseconds per request once the gateway was deployed in the same region as our services. The larger costs were organisational. Someone has to own the gateway’s configuration, and the routing rules become a thing that can be wrong. We had one incident where a fallback rule silently sent traffic to a model that was cheaper but noticeably worse, and nobody noticed for two days because the requests were succeeding.
That incident is why evaluation now sits beside the gateway. Routing changes are tested against a fixed set of real prompts before they ship.
Choosing one
The category is crowded and the feature lists look similar. Four things separated the options for us:
| Criterion | What to check |
|---|---|
| Overhead | Measure p99 latency added at your real request rate, not the vendor’s number. |
| Provider coverage | Every provider you use today and the two you are evaluating. |
| Governance | Per-key budgets, rate limits, and an audit log that survives a restart. |
| Deployment | Self-hosted in your own network, or managed. Data residency usually decides this. |
We shortlisted three, ran a week of shadow traffic through each, and chose on overhead and governance. The exact product matters less than the discipline of testing it with your own traffic before you commit.
Questions readers ask
Does an AI gateway add latency?
A well-built one adds a few milliseconds of overhead per request. The bigger latency risk is a gateway hosted far from your application or the model provider, so deploy it close to both.
Do I need a gateway if I only use one provider?
Probably not on day one. The moment you add a second provider, a fallback model or a per-team budget, a gateway is easier than reimplementing those features in every service.