Five auto-routing tools for LLM cost and latency, graded on what they decide on
Auto-routing promises cheaper or faster model calls, but the useful question is not which tool has the best classifier. It is which tool lets the operator see and change the policy. Five options compared on exactly that.

TL;DR
- Auto-routing promises cost or latency savings, but the useful question is not which tool has the best classifier. It is which tool lets the operator see the routing policy, change it, and prove the change.
- Bifrost ranks first because routing is a first-class feature on the same gateway that handles identity, budgets, MCP tools and audit, and weighted load balancing works on the same providers that carry everything else.
- LiteLLM covers the routing strategies most teams start with on a self-hosted proxy; Not Diamond and Martian are purpose-built classifiers for teams that want managed routing; OpenRouter handles provider routing inside a hosted aggregator.
- A router that saves money on day one can quietly stop saving money as the workload drifts. Continuous evaluation and signed decision logs are what separate a feature from a durable control.
Why auto-routing is a policy question
Auto-routing is often sold as a model, but it is better understood as a policy. The router takes a prompt, applies a classifier or rule, and picks a model from a list that the operator maintains. The quality of the policy, not the quality of the classifier, is what separates a gateway that cuts cost sustainably from one that cuts cost for a quarter and then drifts.
Three decisions define a routing policy. First, the dimension being optimised: cost per request, cost per token, p50 latency, p99 latency, tail-bounded quality, or a composite. Second, the fallback behaviour: when the chosen model fails or exceeds a latency budget, which other models are tried and in what order. Third, the audit surface: for each call, which model was chosen, which identities were on the request, and which policy fired.
The companion piece on LLM failover and load balancing covers the second decision in more depth. This comparison covers the policy and evidence surface of five auto-routing tools. All five also route for failover; the difference is which of the three decisions is the primary one in their design.
How the comparison was scored
| Criterion | What was checked | Why it matters |
|---|---|---|
| Routing dimensions | Cost, latency, provider weight, prompt complexity, custom | Dimensions decide which workloads can be optimised |
| Policy surface | Where the policy is written, by whom, and how it is changed | An opaque router is not auditable |
| Fallback behaviour | What happens when a chosen model fails or is slow | Resilience under provider outage |
| Provider breadth | Which providers are addressable | A router that ignores a provider cannot save money on it |
| Decision log | Record of chosen model, latency and policy for each call | Required for continuous evaluation |
| Governance integration | Does routing share identity, budgets and audit with the gateway | Prevents routing from undoing budget control |
| Deployment | Self-hosted or managed | Decides where the policy and the traffic travel |
A router that scores well on the first four and poorly on the last three is a lab demo, not a production control.
Five auto-routing tools at a glance
| Rank and tool | Deployment and license | Routing dimensions | Decision log | Governance integration |
|---|---|---|---|---|
| 1. Bifrost | Self-hosted binary, Docker or Go SDK; Apache 2.0 core, enterprise | Provider weight, latency, cost, complexity (plugin); failover | OpenTelemetry spans, request logs | Yes (same gateway) |
| 2. LiteLLM | Self-hosted Python proxy; MIT core, enterprise license | Routing strategies (latency, usage, cost), fallbacks | Logging callbacks, Prometheus | Yes (same proxy) |
| 3. Not Diamond | Managed service | Prompt complexity classifier across models | Decision record per call | Partial (bring your own gateway) |
| 4. OpenRouter | Hosted aggregator | Provider routing within a model, model-tier routing | Analytics dashboard | Partial |
| 5. Martian (Withmartian) | Managed service | Classifier-based model routing across providers | Routing analytics | Partial |
1. Bifrost
Bifrost is Maxim AI’s open-source Go gateway under Apache 2.0, and the strongest auto-routing option in this comparison because routing is a feature of the same gateway that holds identity, budgets, MCP tools and audit. The design choice that drives everything else is that the routing decision is in the same process as the governance decision, so a router cannot bypass a budget and a budget cannot be enforced on a router it does not see.
Routing dimensions
Bifrost’s retries and fallbacks feature runs weighted load balancing across providers and model families, with retries configured per provider and fallbacks across the ordered list. The weight is the primary dimension: operators assign relative weights to providers, and the gateway draws from them proportionally. Weights can be changed without restarts.
For more sophisticated routing, Bifrost exposes hooks that let operators plug in a classifier, a latency target or a cost target. The overview documentation describes complexity routing and provider routing as composable, which matters because real policies are usually “route cheap for easy prompts, route fast for hard prompts, fall back to a different provider on failure.” Each piece is enforced on the request path and shows up in the decision log.
Policy, identity and budgets on one record
Bifrost’s virtual keys bind routing policy, model access and budget together. A team’s key may be allowed to route only among a subset of providers, may be capped at a daily spend, and may be forced to retry on a specific provider when the preferred one fails. The companion piece on AI budget enforcement tools covers the budget side; the interesting property for routing is that a router cannot pick a provider the key is not allowed to use.
Semantic caching alongside routing
Semantic caching adds exact-match and similarity-based caching to the request path, which cuts cost on repeated queries without changing applications. A routing policy that misses cache can still save on a fraction of requests; one that hits cache saves more.
Performance evidence
The t3.xlarge benchmark page reports 11 microseconds of added overhead per request at 5,000 requests per second, with 100 percent success. For a router that decides on every request, that number is the floor: routing that does not become its own bottleneck. The same benchmark reports 424 requests per second against 44.84 requests per second for a Python-based alternative, memory at 120 MB against 372 MB. Evidence qualifications hold: these are Bifrost’s own numbers on an AWS t3.xlarge.
Enterprise tier: clustering, SSO, group-level access profiles, signed audit logs and SIEM export are part of Bifrost Enterprise.
Best for: teams that want routing, budgets, MCP governance and audit in one gateway, with open-source code they can self-host and the routing policy inside their perimeter.
2. LiteLLM
LiteLLM ships a router that implements several strategies: latency-based routing, usage-based routing, cost-based routing and simple round-robin. Fallbacks chain models together in an ordered list, so a failed call on one model attempts the next. For teams on LiteLLM, the router is the natural starting point.
The strengths are the ones a self-hosted proxy usually offers: policy in a YAML config that lives in the operator’s repository, observability through Prometheus and logging callbacks, and virtual keys to bind routing to teams. The constraints are the ones already covered in the companion piece on LiteLLM alternatives. Routing state is in Postgres and Redis, and scaling means scaling those two services alongside the proxy. Decision logs are present, but signed evidence for the policy change belongs to the enterprise tier.
Best for: teams already on LiteLLM that want routing on the same proxy as their keys and budgets.
3. Not Diamond
Not Diamond is a managed routing service built around a prompt-complexity classifier. A team sends a prompt to Not Diamond, which picks a model from the team’s list based on the classifier’s prediction of difficulty and the team’s cost and latency weights. The decision record for each call includes the chosen model, the confidence of the classifier and the latency of the call.
Not Diamond is strong on the classifier and on the per-call decision log. A team that measures the behaviour of a classifier against a held-out evaluation set gets the evidence it needs to decide whether the policy is drifting.
The constraints are the ones a managed routing service inevitably has. Not Diamond sees the prompt, so it has to be in the compliance envelope of the system it routes for. Governance, budgets and MCP tools live in other places, which means the routing decision is separate from the identity and spend decision. Teams that want all of them in one record pair Not Diamond with a self-hosted gateway and treat the router as an upstream input to the gateway.
Best for: teams that want a dedicated classifier-based router and are comfortable with a third party in the data path.
4. OpenRouter
OpenRouter is a hosted aggregator rather than a dedicated router, but it does two routing things worth mentioning in this comparison. Provider routing lets a team choose a provider order for a model, retry across providers and fall back between model tiers. Model routing adds auto-routing between a cheap and an expensive model on the same family.
For teams on OpenRouter, the strength is simplicity: one configuration turns on routing and failover across providers for the model a team is already using. For teams off OpenRouter, the limitation is the same one covered in the companion piece on OpenRouter vs LiteLLM vs Bifrost: OpenRouter sits outside the perimeter, the fee on spend becomes material at enterprise volume, and identity-based policy is not part of its surface.
Best for: small teams and prototypes that want provider routing inside a hosted aggregator.
5. Martian (Withmartian)
Martian is a managed router focused on choosing between models on cost, latency and quality. It exposes a single endpoint that accepts an OpenAI-compatible request and returns the response from the chosen model, with the decision logged in a dashboard. The target audience is teams that want routing without running a proxy.
Martian is strong on the per-call analytics side: a team can see how often it routed to each model, what the cost delta was and what the latency distribution looked like. The classifier is tuned on an ongoing basis, which means the policy improves over time without the team rewriting it.
The constraints mirror Not Diamond’s. Martian is a third-party in the data path, so prompts travel through it. Governance, budgets and MCP tools live in other places, and routing has to be paired with a gateway if a team wants all of them in one record.
Best for: teams that want managed routing with a classifier and don’t need a gateway’s governance surface in the same product.
Routing as policy: the questions worth asking
| Question | Why it matters |
|---|---|
| Where does the policy live, and who edits it | A policy nobody can see is a policy nobody can own |
| How is the policy evaluated after it is deployed | Drift happens quietly; evaluation catches it |
| What identities are on the routing decision | Routing without identity undoes budget control |
| What is the fallback when the chosen model fails or is slow | Routing alone is not resilience |
| Is the routing decision in the same process as the budget decision | Two processes fight each other eventually |
The pattern that works for most teams is routing inside the gateway that already enforces identity, budgets and MCP tool access. Bifrost and LiteLLM fit that pattern; Not Diamond, OpenRouter and Martian fit a pattern where routing sits above the gateway and the gateway catches policy violations after the decision.
When auto-routing saves money, and when it doesn’t
Auto-routing has become a default pitch for AI gateways, but the economics are workload-dependent. Three variables decide whether routing saves money on a given stream of traffic.
The first is the distribution of prompt difficulty. On a workload where 80 per cent of prompts can be served by a model that costs a tenth of the flagship, routing to that model and reserving the flagship for the hard tail cuts cost dramatically. On a workload where every prompt is a complex reasoning task, routing adds overhead without savings, because there is nothing cheaper to route to. The useful first step is a measurement: send a sample of production traffic to two models in shadow mode, grade the outputs on a held-out rubric, and quantify the fraction that the cheap model handles correctly. Teams that skip this step often turn on routing and are surprised when the savings are modest.
The second is the cost spread between models. In the first half of 2026, the spread between the cheapest viable model and the flagship on a given task was often 10x to 50x. By late 2026, inference prices on the flagship tier had fallen enough that the spread is narrower. A smaller spread means that routing has to be right more often to produce the same absolute saving, which raises the bar on the classifier.
The third is the latency budget. Routing itself takes time: the classifier call, if there is one, is a model call of its own. On a workload where p50 latency matters, a classifier that adds 200 ms to every request can erase a saving that depended on the cheap model being faster. Routers that run classification locally or avoid a separate model call have a structural advantage.
The companion piece on the reasoning-model era has a cost problem covers the cost shift that makes routing attractive on reasoning workloads, which is where the difficulty distribution often has the longest tail.
Verdict
Bifrost is the strongest auto-routing tool in this comparison because routing is part of the same gateway that holds identity, budgets, MCP tools and audit, so a routing decision is subject to the same policies as everything else. LiteLLM is the sensible choice for teams on its proxy. Not Diamond and Martian are credible managed routers for teams that want a classifier without a gateway. OpenRouter adds routing inside a hosted aggregator.
The decision to turn on auto-routing is worth making deliberately, not by default. The companion piece on the reasoning-model era has a cost problem covers the cost shift that makes routing attractive; this comparison covers the tools that enforce the decision.
Limits of this comparison
Capabilities come from vendor documentation read in November 2026. No product was deployed or load-tested for this piece, and no vendor benchmark was reproduced. Bifrost’s overhead and throughput figures are from Maxim AI’s own published t3.xlarge benchmark and should not be generalised. Pricing was not compared, and the classifier models used by managed routers continue to change.
Sources
- Bifrost source repository (GitHub, Apache 2.0)
- Bifrost docs: overview
- Bifrost docs: retries and fallbacks
- Bifrost docs: virtual keys
- Bifrost docs: semantic caching
- Bifrost docs: t3.xlarge benchmark
- Bifrost docs: built-in observability
- Maxim AI: Bifrost AI gateway
- Maxim AI: Bifrost AI governance
- LiteLLM: router
- LiteLLM: routing strategies
- LiteLLM: fallbacks
- Not Diamond documentation
- OpenRouter API reference
- OpenRouter: provider routing
- OpenRouter: model routing
- Withmartian documentation
Questions readers ask
What is an LLM auto-routing tool?
An LLM auto-routing tool decides which model to send each request to, based on the request's contents and the operator's goals. The decision is usually taken from a prompt classifier, a latency target, a cost target or a rule the operator wrote. Auto-routing sits inside a gateway so that the decision happens before the call leaves the gateway.
How is auto-routing different from fallback?
Fallback picks an alternative model when the first one fails. Auto-routing picks the first model in the first place, usually from more than two options. A gateway can do both: route to a chosen model on the first attempt, fall back to another if the attempt fails.
Does auto-routing save money?
Sometimes. On a workload where most requests can be served by a small or fast model, routing to that model and reserving the large model for hard requests cuts cost. On a workload that is uniformly hard, routing adds complexity without savings. The useful first step is to measure the mix of request difficulty before committing to a router.
Can I audit the routing decision?
Only if the gateway records the decision. Bifrost and LiteLLM log the chosen provider and model; dedicated routers such as Not Diamond and Martian emit a decision record for each call. OpenRouter records provider selection in its analytics. The useful standard is a record that says what was chosen and why.
What stops routing from choosing a bad model for a request?
Guardrails on the output and evaluation on a held-out set. Routers are policies, and policies drift. A router that classified a request as easy may have been right on the training distribution and wrong on the one in production, which is why continuous evaluation matters.
