A Rust beta vs an old Bifrost build: the version problem in software benchmarking

Software benchmarking measures builds, not brands. A result describes one version of a system, with one configuration, called on one route, on one machine, and it stops describing anything else the moment any of those change. LiteLLM’s Rust gateway benchmark is a clear case: it set an early Rust beta, which LiteLLM’s own blog says is still landing streaming, against Bifrost v1.6.4, an older build of the open-source gateway, and the homepage that carries the headline multipliers does not show the Bifrost version.

TL;DR

• LiteLLM’s benchmark blog lists the versions as the LiteLLM Rust beta, LiteLLM Python v1, Bifrost v1.6.4 and the current open-source build of a fourth gateway; the homepage shows no Bifrost version at all.
• The Rust gateway is, in LiteLLM’s words, “an early beta, streaming and the full feature surface are still landing”, and LiteLLM’s docs say Chat Completions and Messages still run on Python in main:latest.
• The gateway’s changelogs list v1.6.5 to v1.6.11 and v2.0.0 to v2.4.0 after the tested build, and 2.x has since reduced memory use, with no figure published yet.
• The version gap does not erase the Rust beta’s real leads: it uses less memory in every file and less CPU, and leads in an exploratory concurrency-256 run.
• For software benchmarking generally, the lesson is to name exact versions where the claim appears; a planned Part 2 rerun is set to use a current build, logging off and on, repeated trials with error bars, and full hardware and worker disclosure.

Why version parity matters in software benchmarking

Version parity means every system in a benchmark is tested at a stated, comparable version, so differences in the results come from the systems and not from where each one sits in its release history. Without it, a benchmark compares a moment in one project’s development with a different moment in another’s, and the headline says nothing about either product as it ships today.

Three things change between releases of infrastructure software, and each one moves benchmark numbers:

• Defaults. A setting that was off in one release can be on in the next, or the reverse. A test that relies on defaults tests whatever the defaults were in that build.
• Feature surface. Betas often lack features that stable releases carry. A system that does less work per request can look faster or smaller for reasons that disappear once the feature lands.
• Memory and resource behavior. Projects regularly change what they cache, buffer or retain. A memory figure from an older build records an implementation that may no longer exist.

Figure 1 shows the chain a benchmark number actually depends on. The product name in a headline sits at one end; the measured result sits at the other, valid only for the specific release line, exact version, configuration and route in between.

Figure 1: Change any link in the chain, from the version to the route, and the measured number describes a different system.

None of this makes old-build or beta benchmarks worthless. A team that runs the exact versions tested can use the numbers directly, and a beta benchmark is a reasonable progress report on a beta. The problem starts when a result moves from a methodology page to a homepage tile, and the version travels with it in neither direction. The Frontier Wire’s analysis of benchmark sizes and statistical noise makes the parallel case for sample size: a score is only as general as the conditions it was measured under.

The versions in LiteLLM’s gateway benchmark

LiteLLM’s benchmark blog names four systems: the LiteLLM Rust beta, LiteLLM Python v1 (litellm[proxy]), Bifrost v1.6.4 and the current open-source build of a fourth gateway. Only one of those is a pinned release number. The Rust entry is a beta, the Python entry gives no version number, and the fourth gateway is identified only as “current”.

The LiteLLM benchmark blog states the setup in one line: “LiteLLM Rust beta, LiteLLM Python v1 (litellm[proxy]), Bifrost v1.6.4, current [fourth gateway] OSS. Same mock and load driver on one host.” Every gateway, it says, forwards the Anthropic Messages body “with no logging callbacks, spend tracking, or persistence enabled”. The files behind the results sit in LiteLLM’s ai-gateway-bench repository. The table records what the blog, the repo and the homepage say about each system, as of November 1, 2026.

Read across, the table shows a benchmark in which the system with the most precise version label is an old build, and the system whose results lead the homepage is a beta. Neither fact appears next to the numbers on litellm.ai.

The Rust beta and its feature surface

The Rust gateway LiteLLM benchmarked is a narrower beta, not the full LiteLLM product rewritten in Rust, and LiteLLM’s own blog says streaming is still landing. A beta that does less work per request is measured doing less work, which matters when its numbers are set against a full release.

LiteLLM is direct about the build’s state. The benchmark blog describes the Rust gateway as “an early beta, streaming and the full feature surface are still landing”, and adds that the setup isolates forwarding overhead and “is not a full-feature comparison”. The consequence follows from that framing: such results cannot establish the performance of the complete production feature set.

LiteLLM’s documentation adds a second version question: what users running LiteLLM today actually get. The Rust gateway docs carry a warning that the Python proxy on main:latest uses Rust for /ocr across providers and for /audio/transcriptions on Bedrock. Other routes, including Messages and Chat Completions, run on Python, and the per-model rust: true flag “currently has no effect”. The same docs also retain older coverage descriptions beneath that warning, so any deployment claim about the Rust path needs a specific version.

The migration tracker shows where the port stands. As of November 1, 2026, it reports 24% of Major APIs built as of v1.107.0-rc.2, a release candidate, measured as 35.4 of 146 weighted engineering points. Router migration, database access, budgets and rate limits, and database-backed configuration are at 0%. The state of the port breaks down as follows:

• Implemented Rust components include HTTP, streaming, tracing, model catalog and config parsing as core infrastructure.
• Partially implemented areas include provider translation, auth, caching, cost calculation and logging, at varying levels.
• Still on Python are authentication, routing, logging, callbacks and spend tracking, which the existing proxy owns while Rust coverage develops.

The presence of those components does not establish an end-to-end Rust request path. For a reader, that leaves three distinct LiteLLM builds in play: the Rust beta that was benchmarked, the release candidate the tracker measures, and main:latest, where Messages traffic is still served by Python. The benchmark speaks to the first. Teams running the Python proxy today will find the operational comparison in The Frontier Wire’s guide to LiteLLM alternatives.

The v1.6.4 build and what has shipped since

Bifrost v1.6.4, the build LiteLLM tested, predates a series of later releases: the project’s changelogs list v1.6.5 to v1.6.11 and then v2.0.0 to v2.4.0. The benchmark therefore measured a release that sits behind a full major version line, launched from a minimal config, and its results describe that build’s defaults and memory behavior.

The repo’s Bifrost gateway README records the version: the benchmark used “Bifrost v1.6.4 via its npx launcher”. The config beside it points a simulated upstream provider at a local port, sets config_store: enabled: false and lists no plugins. It never sets logs_store or enable_logging. The gateway’s public changelog shows how far the release line has moved, with v2.4.0 at the top as of November 1, 2026, and entries such as v2.4.0 give the command to pin that exact transport version through npx or Docker.

Version matters to the configuration finding as well as to the results. The unset logging fields appear to leave logging on: enable_logging defaults to true when unset, a missing logs_store creates a default SQLite log store, and the built-in logging plugin loads regardless of the empty plugin list. That reading is inferred from v2.4.0 code and not yet confirmed on v1.6.4, the version tested, so the version gap that complicates LiteLLM’s benchmark also limits any critique of its configuration. The LiteLLM Rust config has not been checked the same way.

Some v1.6.4 behavior can be read directly at the tagged source. In v1.6.4’s log store configuration, the writer flushes at 1,000 entries, 5 seconds from the first entry or 300 MiB of estimated payload, whichever comes first, with a 10,000-entry queue. That puts queue-plus-batch capacity at approximately 11,000 entries, with new entries dropped when the queue is full; these are entry-count bounds, not a total memory cap. The v1.6.4 SQLite setup opens a disk-backed logs.db with _cache_size=10000, a suggested limit of 10,000 pages per connection, approximately 39.1 MiB at a 4 KiB page size. That cache grows on demand under SQLite’s cache semantics; it is not preallocated, and the files do not quantify its contribution.

Figure 2 places the tested build on the release line.

Figure 2: The comparison paired a pre-release Rust gateway with a Bifrost build that later 1.6 and 2.x releases have since replaced.

Where the version is shown, and where it is not

The Bifrost version appears in LiteLLM’s benchmark blog and in the repo’s gateway README, but not on the homepage that carries the headline figures. The homepage shows an old build’s results with the version hidden. A reader who sees only the tiles has no way to know which build produced the numbers.

The homepage presents a “~22 MB memory at rest” tile, a 0.66 ms p99 added-latency figure and “~4.5× more requests per dollar”, alongside the phrases “Identical hardware” and “Run it yourself”. Its “See the benchmark” link goes to the in-page anchor #benchmark, not to raw data or a methodology document. The blog, in contrast, carries the version list and its own caveats: single-host, per-scenario runs “without repeated-trial error bars”, to be treated “as order-of-magnitude differences”, in “a vendor-run benchmark”. The homepage tiles do not repeat those caveats.

The pattern is common in vendor benchmarking. A methodology page with versions and caveats is written for engineers; the summary tile is written for buyers, and the details rarely make the move. The result is a claim that is correct about one build on one page and unqualified on another. The same gap shows up in other parts of the benchmark. The homepage’s “memory at rest” label is mislabeled: the underlying column is peak RSS, and the homepage uses the widest of three memory pairs in the files. The Frontier Wire’s claim-by-claim fact check covers those rows in full.

For version disclosure specifically, the fix is small. A version string beside each tile, a link to the exact config used, and a date would let a reader decide whether the result applies to the build they run.

The native Anthropic messages route

Bifrost v1.6.4 was benchmarked on its native /anthropic/v1/messages route, because, according to the repo’s gateway README, that build does not mount the Anthropic integration at the root /v1/messages path. The benchmark records that as a native-prefix limitation of the build and does not use a proxy rewrite to work around it.

The README states it directly: the Anthropic-compatible provider points at the local simulated upstream and “is benchmarked through Bifrost’s native /anthropic/v1/messages route using the anthropic/mock model name”. The gateway’s current Anthropic SDK integration docs list /anthropic as the endpoint prefix and show clients setting their base URL to the gateway address plus /anthropic.

The route detail matters to version parity in two ways. First, routes are version-specific: which paths a build mounts, and how it translates between request formats, is exactly the kind of thing that changes between releases. A benchmark that records the route used for each gateway is giving the reader information they need to reproduce it. Second, different routes can run different code paths inside the same gateway. When systems are compared on different paths, the comparison includes whatever each path does, and a reader rerunning the test needs to call the same path to get the same result. The route belongs alongside the other setup gaps: no hardware, worker or core counts are stated for any gateway’s runs.

What the version gap does and does not explain

The version gap means the results describe a Rust beta and Bifrost v1.6.4 under the tested configurations, not the current builds of either project, but it does not overturn the Rust beta’s real leads. The leads are real: the Rust beta uses less memory in every file, less CPU, and leads at concurrency 256.

Those concessions belong in any honest reading of the version problem. As of November 1, 2026, the files show:

These results describe the tested configurations; they do not establish an equivalent-feature performance advantage. That is the version problem stated as a caveat. The memory gap is the clearest case. In cpu_bifrost.csv, the v1.6.4 build’s RSS is already about 282 MB before the first request, and load adds only about 16 MB. That points primarily to a baseline footprint rather than log-buffer growth, while noting that RSS alone cannot identify which components account for it. A baseline footprint is a property of a build, and builds change.

On latency and throughput, the version question is less pressing because the two systems were already close. In LiteLLM’s sweep, the v1.6.4 build adds less p99 latency at concurrency 1 (+0.75 vs +0.89 ms) and 64 (+28.68 vs +40.28 ms), and throughput at concurrency 64 is 2,722.2 RPS for Rust and 2,719.1 for the older build. The Frontier Wire’s latency analysis walks through the sweep row by row, and the hub report on the benchmark covers all three headline claims.

Bifrost 2.x and the memory question

Bifrost 2.x has since reduced memory use by changing what data it retains and how it stores it in memory, and the v1.6.4 benchmark does not reflect those changes. No figure has been published for the reduction; the planned Part 2 rerun is meant to quantify it on a current build.

That statement is the reason version parity matters most for this benchmark. The headline memory pair, 199.1 MB against 21.8 MB, and the two other pairs in the files, 144 vs 26.1 MB and 282 to 298 vs 35 to 36 MB, all sit in a benchmark whose stated Bifrost version is v1.6.4, though the pairs may come from different scenarios or builds. The memory figures describe the build the benchmark names, and the project has since changed how its current release line uses memory.

Two limits keep this from becoming a counter-claim. First, a reduction without a number cannot be compared with anything, so it does not close or narrow the gap on paper; it only says the measured build is no longer current. Second, Rust’s memory advantage in the files is not something the version gap argues away. The files do not establish how much, if any, of that gap comes from logging buffers; answering it requires profiling and a logging-off comparison. The honest position, as of November 1, 2026, is that the memory figures are accurate for v1.6.4, that the current release line behaves differently by the project’s own account, and that the size of the difference is unpublished.

Readers who want the vendor’s own account of how the gateway compares on features can read a feature-by-feature LiteLLM comparison and a piece on high-throughput workloads. Both are vendor material, and both are best read with the same version questions in mind.

What the Part 2 rerun will change

The planned Part 2 rerun commits to repeating the comparison on a current Bifrost build, with logging off and on, repeated trials, error bars and full disclosure of the test setup. Each commitment answers a gap in the published benchmark, and the first two address the version problem directly.

There are four commitments:

• A current Bifrost build, with repeated trials and error bars, in place of single runs on v1.6.4.
• Logging off, and again with logging on, for both gateways, so the configuration gap is measured rather than inferred.
• Full disclosure: hardware, worker counts, simulated-upstream latency and concurrency published for every run.
• A heap profile to measure how much of the memory gap is the log buffer.

The switches a rerun would use are documented. client_config.enable_logging: false removes the built-in logging plugin’s queue and hooks at startup, while other plugins remain enabled independently; client_config.disable_content_logging: true reduces logged payloads. Lowering the writer to max_batch_size: 100, batch_interval: "1s" and write_queue_capacity: 1000 brings capacity down to approximately 1,100 entries, which is not necessarily 10x lower RSS and can increase write overhead and log drops under bursts. The gateway’s observability documentation describes enable_logging as the master toggle for request tracing.

What Part 2 has not promised is just as useful to note. It has not committed to testing a newer LiteLLM Rust build; the version on that side of the comparison is whatever LiteLLM ships. Until Part 2 publishes, no current-build performance figure is available. Teams evaluating a move between gateways can read the vendor’s migration guide and its LiteLLM migration resource, with the caveat that a vendor guide is not a benchmark.

A version-disclosure checklist for readers

A reader can check any software benchmark for version parity with ten questions, from whether exact versions are named to whether the run would be repeated on the builds a team would deploy. Each item below names what to look for and why it matters; LiteLLM’s benchmark serves as the worked example.

Figure 3 condenses the checklist into the four checks that matter most.

Figure 3: A result that fails any version check can still be accurate, but only for the exact builds in the test.

A short version of the same list, for use when reading any vendor benchmark:

1. Find the version string for every system. If the page with the headline does not show it, look for it in the methodology, and treat its absence there as a gap.
2. Check whether each system is a beta, a release candidate or a stable release, and whether the stages match.
3. List the features each system has switched on, especially streaming, logging and persistence.
4. Confirm the configuration file and the defaults for that exact version, not a later one.
5. Note the route each system was called on and whether a client would call the same one.
6. Compare the tested version with the current release, using the project’s changelog.
7. Look for hardware, worker counts, repeated trials and error bars.
8. Ask whether the vendor has committed to rerunning when versions move.

The Frontier Wire’s survey of open-source AI gateways applies the same habits at category level, and the vendor’s own buyer’s guide to LLM gateways is one more source to hold to the checklist.

What the version problem means for teams

For teams choosing a gateway, the version problem means LiteLLM’s benchmark supports a narrower claim than its homepage implies: an early Rust beta, without streaming, ran within a fraction of a millisecond of an older Bifrost build at concurrency 1 to 16, with throughput gaps under 1%, and used less CPU and memory. It does not describe either project’s current release.

That reading is fair to both sides. The Rust beta’s resource leads are real in the files, and nothing in the version gap undoes them. The v1.6.4 build’s memory figures are accurate for that build, and the project’s own account is that 2.x changed the behavior behind them, without a number yet. Both systems have moved since the test, and LiteLLM’s own docs say the Rust path is not yet what main:latest users get for Messages and Chat Completions.

The practical step is the one most software benchmarking guidance arrives at: test the versions you would deploy, on the routes your clients call, with the features you would turn on, and record all three. A benchmark run that way ages slowly, because its versions are on the page and a rerun is a matter of changing one string. LiteLLM’s repository made this review possible by publishing its files, and Bifrost’s source is public in its GitHub repository including the v1.6.4 tag. As of November 1, 2026, the version question in this benchmark is open on both sides, and Part 2’s current-build rerun, with published hardware and error bars, is the test that would answer it.

Source: Debunking LiteLLM Rust Benchmarking: Part 1, by Akshay Deo of Maxim AI, which builds the Bifrost gateway.