Independent reporting on artificial intelligence.


The Frontier Wire

Comparisons

Five auto-routing tools for LLM cost and latency, graded on what they decide on

Auto-routing promises cheaper or faster model calls, but the useful question is not which tool has the best classifier. It is which tool lets the operator see and change the policy. Five options compared on exactly that.

The Frontier Wire cover for a 2026 comparison of five LLM auto-routing tools graded on how they decide cost and latency trade-offs.
The Frontier Wire cover for a 2026 comparison of five LLM auto-routing tools graded on how they decide cost and latency trade-offs.

TL;DR

  • Auto-routing promises cost or latency savings, but the useful question is not which tool has the best classifier. It is which tool lets the operator see the routing policy, change it, and prove the change.
  • Bifrost ranks first because routing is a first-class feature on the same gateway that handles identity, budgets, MCP tools and audit, and weighted load balancing works on the same providers that carry everything else.
  • LiteLLM covers the routing strategies most teams start with on a self-hosted proxy; Not Diamond and Martian are purpose-built classifiers for teams that want managed routing; OpenRouter handles provider routing inside a hosted aggregator.
  • A router that saves money on day one can quietly stop saving money as the workload drifts. Continuous evaluation and signed decision logs are what separate a feature from a durable control.

Why auto-routing is a policy question

Auto-routing is often sold as a model, but it is better understood as a policy. The router takes a prompt, applies a classifier or rule, and picks a model from a list that the operator maintains. The quality of the policy, not the quality of the classifier, is what separates a gateway that cuts cost sustainably from one that cuts cost for a quarter and then drifts.

Three decisions define a routing policy. First, the dimension being optimised: cost per request, cost per token, p50 latency, p99 latency, tail-bounded quality, or a composite. Second, the fallback behaviour: when the chosen model fails or exceeds a latency budget, which other models are tried and in what order. Third, the audit surface: for each call, which model was chosen, which identities were on the request, and which policy fired.

The companion piece on LLM failover and load balancing covers the second decision in more depth. This comparison covers the policy and evidence surface of five auto-routing tools. All five also route for failover; the difference is which of the three decisions is the primary one in their design.

How the comparison was scored

CriterionWhat was checkedWhy it matters
Routing dimensionsCost, latency, provider weight, prompt complexity, customDimensions decide which workloads can be optimised
Policy surfaceWhere the policy is written, by whom, and how it is changedAn opaque router is not auditable
Fallback behaviourWhat happens when a chosen model fails or is slowResilience under provider outage
Provider breadthWhich providers are addressableA router that ignores a provider cannot save money on it
Decision logRecord of chosen model, latency and policy for each callRequired for continuous evaluation
Governance integrationDoes routing share identity, budgets and audit with the gatewayPrevents routing from undoing budget control
DeploymentSelf-hosted or managedDecides where the policy and the traffic travel

A router that scores well on the first four and poorly on the last three is a lab demo, not a production control.

Five auto-routing tools at a glance

Rank and toolDeployment and licenseRouting dimensionsDecision logGovernance integration
1. BifrostSelf-hosted binary, Docker or Go SDK; Apache 2.0 core, enterpriseProvider weight, latency, cost, complexity (plugin); failoverOpenTelemetry spans, request logsYes (same gateway)
2. LiteLLMSelf-hosted Python proxy; MIT core, enterprise licenseRouting strategies (latency, usage, cost), fallbacksLogging callbacks, PrometheusYes (same proxy)
3. Not DiamondManaged servicePrompt complexity classifier across modelsDecision record per callPartial (bring your own gateway)
4. OpenRouterHosted aggregatorProvider routing within a model, model-tier routingAnalytics dashboardPartial
5. Martian (Withmartian)Managed serviceClassifier-based model routing across providersRouting analyticsPartial

1. Bifrost

Bifrost is Maxim AI’s open-source Go gateway under Apache 2.0, and the strongest auto-routing option in this comparison because routing is a feature of the same gateway that holds identity, budgets, MCP tools and audit. The design choice that drives everything else is that the routing decision is in the same process as the governance decision, so a router cannot bypass a budget and a budget cannot be enforced on a router it does not see.

Routing dimensions

Bifrost’s retries and fallbacks feature runs weighted load balancing across providers and model families, with retries configured per provider and fallbacks across the ordered list. The weight is the primary dimension: operators assign relative weights to providers, and the gateway draws from them proportionally. Weights can be changed without restarts.

For more sophisticated routing, Bifrost exposes hooks that let operators plug in a classifier, a latency target or a cost target. The overview documentation describes complexity routing and provider routing as composable, which matters because real policies are usually “route cheap for easy prompts, route fast for hard prompts, fall back to a different provider on failure.” Each piece is enforced on the request path and shows up in the decision log.

Policy, identity and budgets on one record

Bifrost’s virtual keys bind routing policy, model access and budget together. A team’s key may be allowed to route only among a subset of providers, may be capped at a daily spend, and may be forced to retry on a specific provider when the preferred one fails. The companion piece on AI budget enforcement tools covers the budget side; the interesting property for routing is that a router cannot pick a provider the key is not allowed to use.

Semantic caching alongside routing

Semantic caching adds exact-match and similarity-based caching to the request path, which cuts cost on repeated queries without changing applications. A routing policy that misses cache can still save on a fraction of requests; one that hits cache saves more.

Performance evidence

The t3.xlarge benchmark page reports 11 microseconds of added overhead per request at 5,000 requests per second, with 100 percent success. For a router that decides on every request, that number is the floor: routing that does not become its own bottleneck. The same benchmark reports 424 requests per second against 44.84 requests per second for a Python-based alternative, memory at 120 MB against 372 MB. Evidence qualifications hold: these are Bifrost’s own numbers on an AWS t3.xlarge.

Enterprise tier: clustering, SSO, group-level access profiles, signed audit logs and SIEM export are part of Bifrost Enterprise.

Best for: teams that want routing, budgets, MCP governance and audit in one gateway, with open-source code they can self-host and the routing policy inside their perimeter.

2. LiteLLM

LiteLLM ships a router that implements several strategies: latency-based routing, usage-based routing, cost-based routing and simple round-robin. Fallbacks chain models together in an ordered list, so a failed call on one model attempts the next. For teams on LiteLLM, the router is the natural starting point.

The strengths are the ones a self-hosted proxy usually offers: policy in a YAML config that lives in the operator’s repository, observability through Prometheus and logging callbacks, and virtual keys to bind routing to teams. The constraints are the ones already covered in the companion piece on LiteLLM alternatives. Routing state is in Postgres and Redis, and scaling means scaling those two services alongside the proxy. Decision logs are present, but signed evidence for the policy change belongs to the enterprise tier.

Best for: teams already on LiteLLM that want routing on the same proxy as their keys and budgets.

3. Not Diamond

Not Diamond is a managed routing service built around a prompt-complexity classifier. A team sends a prompt to Not Diamond, which picks a model from the team’s list based on the classifier’s prediction of difficulty and the team’s cost and latency weights. The decision record for each call includes the chosen model, the confidence of the classifier and the latency of the call.

Not Diamond is strong on the classifier and on the per-call decision log. A team that measures the behaviour of a classifier against a held-out evaluation set gets the evidence it needs to decide whether the policy is drifting.

The constraints are the ones a managed routing service inevitably has. Not Diamond sees the prompt, so it has to be in the compliance envelope of the system it routes for. Governance, budgets and MCP tools live in other places, which means the routing decision is separate from the identity and spend decision. Teams that want all of them in one record pair Not Diamond with a self-hosted gateway and treat the router as an upstream input to the gateway.

Best for: teams that want a dedicated classifier-based router and are comfortable with a third party in the data path.

4. OpenRouter

OpenRouter is a hosted aggregator rather than a dedicated router, but it does two routing things worth mentioning in this comparison. Provider routing lets a team choose a provider order for a model, retry across providers and fall back between model tiers. Model routing adds auto-routing between a cheap and an expensive model on the same family.

For teams on OpenRouter, the strength is simplicity: one configuration turns on routing and failover across providers for the model a team is already using. For teams off OpenRouter, the limitation is the same one covered in the companion piece on OpenRouter vs LiteLLM vs Bifrost: OpenRouter sits outside the perimeter, the fee on spend becomes material at enterprise volume, and identity-based policy is not part of its surface.

Best for: small teams and prototypes that want provider routing inside a hosted aggregator.

5. Martian (Withmartian)

Martian is a managed router focused on choosing between models on cost, latency and quality. It exposes a single endpoint that accepts an OpenAI-compatible request and returns the response from the chosen model, with the decision logged in a dashboard. The target audience is teams that want routing without running a proxy.

Martian is strong on the per-call analytics side: a team can see how often it routed to each model, what the cost delta was and what the latency distribution looked like. The classifier is tuned on an ongoing basis, which means the policy improves over time without the team rewriting it.

The constraints mirror Not Diamond’s. Martian is a third-party in the data path, so prompts travel through it. Governance, budgets and MCP tools live in other places, and routing has to be paired with a gateway if a team wants all of them in one record.

Best for: teams that want managed routing with a classifier and don’t need a gateway’s governance surface in the same product.

Routing as policy: the questions worth asking

QuestionWhy it matters
Where does the policy live, and who edits itA policy nobody can see is a policy nobody can own
How is the policy evaluated after it is deployedDrift happens quietly; evaluation catches it
What identities are on the routing decisionRouting without identity undoes budget control
What is the fallback when the chosen model fails or is slowRouting alone is not resilience
Is the routing decision in the same process as the budget decisionTwo processes fight each other eventually

The pattern that works for most teams is routing inside the gateway that already enforces identity, budgets and MCP tool access. Bifrost and LiteLLM fit that pattern; Not Diamond, OpenRouter and Martian fit a pattern where routing sits above the gateway and the gateway catches policy violations after the decision.

When auto-routing saves money, and when it doesn’t

Auto-routing has become a default pitch for AI gateways, but the economics are workload-dependent. Three variables decide whether routing saves money on a given stream of traffic.

The first is the distribution of prompt difficulty. On a workload where 80 per cent of prompts can be served by a model that costs a tenth of the flagship, routing to that model and reserving the flagship for the hard tail cuts cost dramatically. On a workload where every prompt is a complex reasoning task, routing adds overhead without savings, because there is nothing cheaper to route to. The useful first step is a measurement: send a sample of production traffic to two models in shadow mode, grade the outputs on a held-out rubric, and quantify the fraction that the cheap model handles correctly. Teams that skip this step often turn on routing and are surprised when the savings are modest.

The second is the cost spread between models. In the first half of 2026, the spread between the cheapest viable model and the flagship on a given task was often 10x to 50x. By late 2026, inference prices on the flagship tier had fallen enough that the spread is narrower. A smaller spread means that routing has to be right more often to produce the same absolute saving, which raises the bar on the classifier.

The third is the latency budget. Routing itself takes time: the classifier call, if there is one, is a model call of its own. On a workload where p50 latency matters, a classifier that adds 200 ms to every request can erase a saving that depended on the cheap model being faster. Routers that run classification locally or avoid a separate model call have a structural advantage.

The companion piece on the reasoning-model era has a cost problem covers the cost shift that makes routing attractive on reasoning workloads, which is where the difficulty distribution often has the longest tail.

Verdict

Bifrost is the strongest auto-routing tool in this comparison because routing is part of the same gateway that holds identity, budgets, MCP tools and audit, so a routing decision is subject to the same policies as everything else. LiteLLM is the sensible choice for teams on its proxy. Not Diamond and Martian are credible managed routers for teams that want a classifier without a gateway. OpenRouter adds routing inside a hosted aggregator.

The decision to turn on auto-routing is worth making deliberately, not by default. The companion piece on the reasoning-model era has a cost problem covers the cost shift that makes routing attractive; this comparison covers the tools that enforce the decision.

Limits of this comparison

Capabilities come from vendor documentation read in November 2026. No product was deployed or load-tested for this piece, and no vendor benchmark was reproduced. Bifrost’s overhead and throughput figures are from Maxim AI’s own published t3.xlarge benchmark and should not be generalised. Pricing was not compared, and the classifier models used by managed routers continue to change.

Sources

  1. Bifrost source repository (GitHub, Apache 2.0)
  2. Bifrost docs: overview
  3. Bifrost docs: retries and fallbacks
  4. Bifrost docs: virtual keys
  5. Bifrost docs: semantic caching
  6. Bifrost docs: t3.xlarge benchmark
  7. Bifrost docs: built-in observability
  8. Maxim AI: Bifrost AI gateway
  9. Maxim AI: Bifrost AI governance
  10. LiteLLM: router
  11. LiteLLM: routing strategies
  12. LiteLLM: fallbacks
  13. Not Diamond documentation
  14. OpenRouter API reference
  15. OpenRouter: provider routing
  16. OpenRouter: model routing
  17. Withmartian documentation

Questions readers ask

What is an LLM auto-routing tool?

An LLM auto-routing tool decides which model to send each request to, based on the request's contents and the operator's goals. The decision is usually taken from a prompt classifier, a latency target, a cost target or a rule the operator wrote. Auto-routing sits inside a gateway so that the decision happens before the call leaves the gateway.

How is auto-routing different from fallback?

Fallback picks an alternative model when the first one fails. Auto-routing picks the first model in the first place, usually from more than two options. A gateway can do both: route to a chosen model on the first attempt, fall back to another if the attempt fails.

Does auto-routing save money?

Sometimes. On a workload where most requests can be served by a small or fast model, routing to that model and reserving the large model for hard requests cuts cost. On a workload that is uniformly hard, routing adds complexity without savings. The useful first step is to measure the mix of request difficulty before committing to a router.

Can I audit the routing decision?

Only if the gateway records the decision. Bifrost and LiteLLM log the chosen provider and model; dedicated routers such as Not Diamond and Martian emit a decision record for each call. OpenRouter records provider selection in its analytics. The useful standard is a record that says what was chosen and why.

What stops routing from choosing a bad model for a request?

Guardrails on the output and evaluation on a held-out set. Routers are policies, and policies drift. A router that classified a request as easy may have been right on the training distribution and wrong on the one in production, which is why continuous evaluation matters.

More comparisons