What an AI gateway does, and what it costs to run one

TL;DR

• An AI gateway is a proxy between applications and model providers. It holds the provider keys and applies routing, fallbacks, budgets, caching and logging in one place instead of in every service.
• The products split by deployment model: self-hosted open-source gateways, managed services, and API-management suites with AI features. In every self-hosted option, the features an enterprise checklist asks for are on the paid tier.
• Bifrost, an Apache 2.0 gateway written in Go, is examined in detail below. Its published benchmark reports 11 microseconds of overhead per request at 5,000 requests per second on an AWS t3.xlarge; clustering, guardrails and single sign-on are enterprise features.

An AI gateway is a proxy that model requests pass through on their way to OpenAI, Anthropic, Google, a cloud provider’s model service or a self-hosted server. The application talks to the gateway; the gateway talks to the provider. In exchange for one extra network hop, a team gets a single API for every provider, a single place to hold credentials, and a single point at which to apply routing, retries, spending limits and logging. The first half of this article covers what that layer does and how the main products differ. The second half goes through one product, Bifrost, in detail, because its documentation is specific enough to check and its core can be installed and run without an account.

Why the layer exists

A company that adopted language models quickly tends to end up with several services calling several providers. Each service has its own hard-coded model names, its own copy of the provider keys and its own retry logic. A price change or an outage at one provider has to be handled in every service separately, and nobody can say what the whole company spent on models last month without assembling a spreadsheet.

A gateway moves that into one process. The features have converged across the category:

• A unified API. Applications speak one request format, usually OpenAI’s, and name a model. The gateway translates to each provider’s native API.
• Fallbacks and retries. When a provider returns an error or times out, the gateway retries and then fails over to a second provider or model.
• Load balancing. Traffic is spread across several API keys or providers, by fixed weights or by measured health.
• Virtual keys and budgets. Each team or application authenticates to the gateway with its own key, which carries a spending limit and a rate limit.
• Caching. Identical or similar requests are answered from a store instead of the provider.
• Observability. Each request’s model, tokens, cost and latency land in one log and one metrics endpoint.

Gartner published a Market Guide for AI Gateways in 2025. In a definition that vendors quote widely, it describes the category as an intermediary between applications and AI services that provides a central point for security, governance and observability. The guide is paywalled, so the wording and the adoption projections attributed to it could not be checked for this article and are not repeated.

How the main products differ

The category divides by where the gateway runs and who operates it. The table covers four representative products, described from their own documentation as read in September 2026.

Two decisions sort most teams. The first is deployment: a team with data-residency obligations, or one that does not want prompts leaving its network, needs a self-hosted gateway, while a team that wants nothing to operate picks a managed one. The second is what is free. Every self-hosted option in the table has a paid tier, and the features on the paid side tend to be the ones a procurement checklist asks for: high availability, single sign-on, audit logs.

Bifrost as a worked example

Bifrost is a Go binary that exposes an OpenAI-compatible HTTP API, a web console and a Prometheus metrics endpoint. The core is published on GitHub under Apache 2.0, with releases shipping weekly. Its documentation lists around thirty providers, including OpenAI, Anthropic, AWS Bedrock, Google Vertex AI and Gemini, Azure OpenAI, Mistral, Cohere, Groq, DeepSeek, Ollama and vLLM. Integration works the way it does for most gateways in the category: an existing OpenAI, Anthropic or Google GenAI SDK client keeps its code and changes its base URL to a path on the gateway, such as /openai or /anthropic, as described in the drop-in replacement docs.

The quickstart gives three ways to run it:

npx -y @maximhq/bifrost
docker run -p 8080:8080 maximhq/bifrost
go get github.com/maximhq/bifrost/core

Providers are configured in the web console, in a config.json file, or through environment variables referenced from that file.

A local install

For this article the npx command was run on an Apple Silicon laptop on 27 September 2026. The wrapper downloaded a 117 MB binary for version 2.2.3 of the HTTP gateway. Once the download finished, the server’s log reported a bootstrap time of under one second and the console answered on port 8080. The startup log lists the plugins active by default: prompts, logging, governance, routing, a compatibility layer, a model-catalog resolver and telemetry. It also synced a library of 124 MCP server entries from the vendor’s site.

With no provider configured, a chat completion request for an OpenAI model returned a structured JSON error naming the missing provider config, along with the provider, the requested model and the request type. The management API refused an unauthenticated request with a 401. No provider keys were added and no live traffic was sent, so the sections that follow describe the documentation, not load testing.

Routing, fallbacks and keys

Three mechanisms in the open-source build decide where a request goes.

Fallbacks are declared per request. The fallbacks documentation describes a fallbacks array of provider and model pairs. The primary provider uses its full retry budget first, then each fallback gets its own. A primary and two fallbacks each configured with three retries can make up to twelve attempts, and the response records which provider served it. Plugins can block the chain when a request has been rejected for a policy reason, so a blocked prompt is not retried elsewhere.

Weighted keys spread traffic across several API keys for one provider. The key management page describes weighted random selection: a key with weight 0.7 receives roughly 70 percent of requests. Keys can be restricted to specific models, and a caller can pin a key by name with a header.

Routing rules are written in the Common Expression Language (CEL), a small expression syntax Google designed for policy. The routing rules documentation shows conditions on headers, team, customer and virtual key, on capacity signals such as the percentage of budget used, and on an automatically assigned complexity tier of simple, medium or complex. Rules are evaluated first-match-wins, scoped from virtual key up to global. This is the mechanism behind routing by difficulty, the pattern in which cheap requests go to a fast model and hard ones to a reasoning model, discussed in the reasoning-model cost report. The documentation publishes no accuracy figures for the complexity classifier, so how well it sorts real traffic is something to test on your own prompts.

Governance with virtual keys

Governance in Bifrost hangs off virtual keys, the credentials an application presents to the gateway. Provider access is deny-by-default: a key can only reach the providers configured for it unless an allow-all flag is set. Models can be allowlisted or blocklisted, including by regular expression.

Each key carries an independent budget with a reset period from one minute to one year, and rate limits by requests or tokens. Keys sit in a three-level hierarchy of customers, teams and keys, so a department budget can cap the keys beneath it. Rate limits apply only at the key level, not to teams or customers; the documentation states this plainly. Keys can expire, after which requests receive a 403.

A gateway does not make the models better. It makes a company’s use of them legible.

This is the feature set that turns a proxy into a control plane, and it is also where the enterprise boundary starts to matter. The identity features that make virtual keys manageable at scale, such as OIDC provisioning, directory sync and role-based access control, are on the paid side.

Caching, observability and MCP

Bifrost’s semantic caching has two paths. Direct matching hashes a normalised request and serves an identical repeat without calling a provider. Semantic matching embeds the request and looks for a stored one above a cosine similarity threshold, which defaults to 0.8. Entries expire after a default of five minutes. Both paths need a vector store, with Redis or Valkey, Weaviate, Qdrant and Pinecone listed, and the semantic path needs an embedding-capable provider. The documentation notes the trade-off: a semantic miss pays for the embedding call on top of the full model call, so it is slower than running with no cache. Teams with little repetition in their traffic should measure hit rates before turning it on.

For observability, the gateway exposes Prometheus metrics at /metrics, including upstream request counts, input tokens, cost totals, time to first token for streams and per-key provider health. An OpenTelemetry plugin exports traces and metrics over OTLP by HTTP or gRPC to collectors such as Grafana Cloud, Datadog, New Relic or Honeycomb. Metrics never carry prompt content, and content logging can be disabled globally, per connector or per request with a header, which matters for anyone whose prompts contain customer data.

Bifrost also acts as a Model Context Protocol (MCP) gateway. It connects to tool servers over STDIO, HTTP or server-sent events, filters which tools each virtual key may see, and can expose the result to clients such as Claude Desktop or Cursor. The documentation claims that a “code mode”, in which the model writes Python to orchestrate tools rather than receiving every tool definition, cuts input tokens by up to 92.8 percent when three or more MCP servers are attached. That is a vendor figure from a vendor scenario and was not reproduced for this article.

The performance claims

Bifrost’s headline number is 11 microseconds of added latency per request at 5,000 requests per second. The benchmarking page gives the conditions: an AWS t3.xlarge with 4 vCPUs and 16 GB of memory, responses of roughly 10 KB, a 100 percent success rate and a peak memory of about 3.3 GB. On a t3.medium with 2 vCPUs, overhead was 59 microseconds. These figures isolate the gateway’s own cost, separate from model latency.

The comparison against LiteLLM is a separate test on a t3.medium at 500 requests per second for 60 seconds. It reports Bifrost at 424 requests per second against LiteLLM at 44.84, a p99 latency of 1.68 seconds against 90.72, 120 MB of memory against 372 MB, and a 100 percent success rate against 88.78 percent. LiteLLM’s own README now describes a Rust core, so a test against an earlier Python-only build may no longer describe the current software. The useful comparison is the one you run against your own traffic, which is also the advice in the evaluation frameworks guide.

What is open source and what is not

The enterprise overview describes Bifrost Enterprise as a strict superset of the open-source gateway with the same configuration schema. The features on the paid side are:

There is no public price list; the vendor offers a 14-day trial. A team that needs high availability, single sign-on or audit trails should budget for a licence rather than assume the open-source build will pass a security review on its own. The same is true of every self-hosted gateway in the category. What differs is where each vendor draws the line.

A separate product sits beside the gateway. Bifrost Edge is an endpoint agent for macOS, Windows and Linux that routes AI traffic from desktop chat apps, browser tools, coding agents and MCP servers on an employee’s machine through the same gateway, so the same virtual keys, budgets, guardrails and audit logs apply. It is in early access. It addresses the shadow-AI problem, and it is too early to assess.

How to evaluate a gateway

Whatever the product, four measurements decide the choice, and the vendor’s numbers are a starting point for none of them.

Run shadow traffic through each shortlisted gateway for a week. Then add one more test: change a routing rule to a cheaper model and check that your evaluation suite notices. A gateway makes it easy to swap models silently, and a fallback that lands on a worse model can succeed on every request while degrading every answer. Routing changes deserve the same regression testing as code.

Limits of this article

Bifrost was run on a laptop with no provider keys, and its documentation was read; it was not run under production load and the enterprise features were not tested. LiteLLM, Cloudflare and Kong were described from their documentation only. Provider coverage, MCP behaviour and the complexity classifier are as the Bifrost documentation described them in September 2026. Documentation changes, and any team relying on a specific figure here should re-check the linked page before acting on it.