GB300 vs H100 in Epoch's inference index: 25x capacity, 2.5x FLOP/s

An Nvidia GB300 can serve roughly 25 times as many simultaneous agent requests as an H100 while doing only about 2.5 times as much 8-bit arithmetic per second. That is the central estimate in Epoch AI’s Inference Performance Index, published on 9 November 2026 by Epoch researcher Jason Li. The gap between the two ratios exists because serving long-context agents is mostly a memory and communication problem. How fast a chip multiplies numbers matters less than how much model and cache it can hold, how quickly it can read them, and how many neighbors it can talk to at high bandwidth.

Two qualifications sit next to that answer. First, the index is modeled, not measured. Its scores are analytical “speed of light” projections from a simulator, run on specification sheets, with a limited check against benchmark data. Second, the 25x is an average over a basket of four models. Using the per-model scores Epoch publishes, GB300’s modeled advantage over H100 is about 8 times on one of those models and about 217 times on another. Of the settings Epoch tested, the choice of basket spans the widest range.

The Frontier Wire did not run the simulator. This comparison reads the launch report, the methodology, the sensitivity and validation studies, and the hardware profiles in Epoch’s public repository. Every multiple below that Epoch does not state itself is our arithmetic from its published score matrix, and is labeled that way.

The short version

• Epoch’s IPI v1 scores GB300 NVL72 at 1 by definition and H100 SXM at 0.040, a modeled 25x per-chip gap on agentic workloads with 262,144 tokens of cached context.
• The hardware profiles give GB300 2.5 times H100’s dense FP8 FLOP/s but 3.6 times the memory, a 72-GPU NVLink domain instead of eight, and FP4 rather than FP8 expert weights.
• Per model, GB300’s modeled advantage over H100 ranges from about 8x (Qwen3-Coder 480B) to about 217x (MAI-Base-2 proxy). For H200 it ranges from about 2.7x to 63x, with 8x on the full basket.
• The headline uses an arithmetic mean, which lets the model Hopper serves best supply most of Hopper’s score. A geometric mean, by our calculation, would put the H100 gap near 59x.
• Validation covers mainly Kimi K3 on GB300, where a half-speed-of-light assumption matched five measured points within 11.6% on average. Hopper is not validated.

The criteria before the verdict

A per-chip inference comparison has to say which resources it counts. Epoch’s model takes five hardware inputs: arithmetic throughput at each supported precision, memory capacity, memory bandwidth, interconnect bandwidth at each tier, and the size and latency of each communication group. The table below lists those inputs as they appear in the index’s frozen hardware profiles for four Nvidia systems, followed by the index score.

The profile values come from the repository’s 15 October 2026 hardware files and its catalog notes, which state that the H100 and H200 figures exclude the sparse tensor-core bonus. They are consistent with Nvidia’s own pages. Nvidia’s H100 page lists 3,958 FP8 teraFLOPS “with sparsity”, which halves to the 1.978 PF dense figure, plus 80 GB and 3.35 TB/s. The GB300 NVL72 page lists 720 FP8 PFLOPS for the 72-GPU rack, with a footnote that tensor-core figures include sparsity unless noted; divided by 72 and halved, that gives 5 PF per GPU.

Read across the rows, GB300’s advantage over H100 is 2.53 times on FP8 arithmetic, 2.39 times on memory bandwidth and 3.6 times on memory capacity. The communication domain is nine times larger. None of those single ratios approaches 25. The index result comes from how they combine on a specific workload, which is why the method matters more than any line of the spec sheet.

Why serving speed is a memory problem

Language model inference has two phases. Prefill processes new input and stores intermediate state in a cache, usually called the key-value or KV cache. Decode produces the response one forward pass at a time, reading the cache and the model weights on every pass. Our guide to self-hosted LLMs covers how the cache grows with context and with the number of requests served at once.

Agents make the cache large. Epoch’s report gives an example of a coding agent sending a request with 200,000 tokens of cached history and 1,000 tokens of new code-search results. The model generates 1,000 output tokens, including a tool call. The next request reuses about 202,000 cached tokens, and only the new file contents need prefill. In the index’s model, that history stays resident in memory and is read again during every decode step.

SemiAnalysis, whose InferenceX benchmark Epoch uses for comparison, describes agentic workloads in its 19 August 2026 overview in the same terms: sessions of tens or hundreds of turns, long contexts and high prefix reuse. OpenRouter’s 8 October newsletter reports that in the last week of September, API keys it classifies as agentic used about seven times the tokens of human keys. It also reports that cached prompt tokens were just under 90 percent of agentic tokens in September. Our prompt caching guide explains the billing side of that pattern.

Batching is how a chip earns its throughput. During decode, weights are read from high-bandwidth memory once per layer and reused across every request in the batch. Each extra request adds its own cache reads, so a bigger batch raises total output while slowing each request. Epoch’s example: moving from two to eight requests per chip lowers output speed from about 210 to 80 tokens per second per request, while throughput rises from about 420 to 640 tokens per second per chip.

That tradeoff is why Epoch does not score chips on a single throughput number. A chip that wins on bulk throughput at slow speeds may lose when each agent must generate quickly. The index therefore measures capacity, meaning simultaneous requests per chip, across a range of required output speeds.

How the index is built

The IPI pipeline has five stages. Each one embeds a choice that a reader could make differently.

Simulate. Epoch runs AISimulate 0.13.0, which it describes as Nvidia’s tool and which is published under the ai-dynamo GitHub organization as the successor to AIConfigurator. The engine is unmodified, pinned to a specific upstream commit. Epoch’s wrappers supply hardware profiles, model adapters, memory rules and the search over layouts. The outputs are speed-of-light (SOL) estimates: for operations with both compute and memory costs, time is the larger of FLOPs divided by FLOP/s and bytes divided by bytes per second. AISimulate’s own README says its tool “does not replace validation on the target hardware.”

Search. For each chip and model, the code searches a finite family of parallelism layouts and every integer batch size that fits in memory. Memory admission uses 90 percent of total HBM, a fixed 4 GiB reservation and model-specific guards. At each required output speed, the densest configuration that meets the speed sets the decode capacity. The methodology calls this “the best of a finite search, not proof of a global optimum.”

Provision prefill. Decode chips cannot run without prefill, so the index adds just enough prefill chips to keep up. If decode supports c requests per chip at speed s, prefill throughput is P new-input tokens per chip-second, and r new-input tokens arrive per output token, combined capacity is c divided by (1 + r × s × c / P). Epoch sets r at 1.513, from a filtered cohort of 110,565 warm-cache Claude coding rounds in a coding-trace dataset.

Integrate over speed. The capacity curve is integrated from 50 to 400 tokens per second per request on a logarithmic axis, so the 50 to 100, 100 to 200 and 200 to 400 bands carry equal weight. The methodology says this is “a deliberate coverage choice, not an estimated probability distribution of demand.”

Normalize and average. Each chip’s area is divided by GB300’s area for the same model, and the four ratios are averaged with equal weights. For Vera Rubin (VR200), the four ratios are 1.225, 2.476, 2.355 and 2.654, which average to 2.177 GB300e. The 2.48 that appears in the report’s figure for Kimi K3 is one of those ratios, not the final score.

A worked example of the prefill adjustment

The inputs here are hypothetical, chosen only to show the arithmetic. Suppose a chip’s best decode layout supports 4 requests per chip at 100 tokens per second, and its prefill throughput is 20,000 new-input tokens per second per chip. Each request then needs 0.25 decode chips and 1.513 × 100 / 20,000, or about 0.0076, prefill chips. Combined capacity is 1 / 0.2576, or about 3.88 requests per chip, roughly 3 percent below the decode-only figure.

The adjustment grows when the speed requirement rises or prefill is slow. Epoch’s sensitivity study shows it barely moves the H100 comparison. Using its published scores, GB300’s multiple over H100 is about 26.2x on decode alone, 25.7x at half the input ratio, 25.2x at the headline ratio and 24.3x at double it.

The workload the index assumes

Version 1 targets agentic inference only. The table collects the assumptions that define it, from Epoch’s methodology and workload record.

The four models were chosen, Epoch says, because AISimulate can model them accurately and because their attention designs differ. GLM-5.2 keeps a compressed cache of the full context but its main attention reads a selected subset of up to 2,048 tokens. Kimi K3 mixes 69 linear-attention layers, whose state stays a fixed size, with 24 full-context layers. Qwen3-Coder 480B uses full-context grouped-query attention throughout. The fourth model is a proxy for Microsoft’s MAI-Base-2, combining sliding-window and global attention. Epoch’s reconstruction has about 2.84 trillion total and 85.9 billion active parameters, and Epoch says it is not Microsoft’s exact implementation.

The trace behind the input ratio is a filtered cohort, not an all-provider average. Rows had to be at least 90 percent cached, with 131,072 to 393,216 cached tokens. The same ratio is applied to all four models, and their different tokenizers and verbosity are not modeled. The underlying trace database is not redistributed.

Where a 25x gap comes from

Epoch does not publish a factor-by-factor decomposition of the H100 result, so this section is our reading of the profiles and model descriptions, not Epoch’s attribution.

Start with what has to fit. The MAI proxy has about 2.84 trillion parameters. At one byte per parameter, which is roughly FP8, that is around 2.8 TB of weights, before any cache. An H100’s high-bandwidth domain is eight GPUs with 80 GB each, or 640 GB. The model cannot fit in one domain, so the index splits it across pipeline stages connected by 50 GB/s links. The methodology says profiles limited to eight-GPU nodes use a “conservative serial” pipeline schedule between stages. A GB300 NVL72 domain holds 72 GPUs with 288 GB each, about 20.7 TB, and the index stores its expert weights in FP4, about half the bytes of FP8.

Then add the cache. Each request carries 262,144 tokens of context. Whatever memory remains after the weights sets how many of those requests fit, and every decode step reads their cache. A chip with 3.6 times the memory can hold many times more requests once weights are subtracted, because the weights take a larger share of a small chip. More requests in memory mean larger batches, and larger batches amortize the weight reads that dominate decode time.

The extremes fit this reading. The gap is largest on the 2.84-trillion-parameter MAI proxy and smallest on Qwen3-Coder 480B, whose roughly 480 GB of 8-bit weights come close to fitting in an eight-H100 domain’s 640 GB, though with little room left for cache. Epoch’s model record puts Qwen’s FP8 cache at about 31 GiB per 262,144-token request, so each extra request in that leftover space is expensive. The same record lists the MAI proxy’s unsharded cache at about 2.3 GiB, which by our arithmetic corresponds to one 262,144-token request. It is small because 85 of the proxy’s 102 layers attend over only a 512-token window. That points to weights and pipeline communication, not cache, as the main burden for the MAI proxy on Hopper. The per-model scores are consistent with this account, but the account itself is inference. Epoch’s sensitivity study holds precision and topology fixed, so it does not isolate how much of the gap comes from FP4 experts or from the serial pipeline schedule.

The B300 comparison helps separate one factor. B300 HGX and GB300 NVL72 use the same Blackwell Ultra GPU, and their profiles differ by about 10 percent on FP8 arithmetic and 3 percent on memory bandwidth. The main difference is an eight-GPU domain versus 72. Epoch scores GB300 NVL72 at about 1.9 times B300 HGX, which the report attributes to “the importance of its larger scale-up domain.”

The basket decides the answer

Epoch’s sensitivity study re-integrates the saved simulation curves under 28 scenarios, including each model alone and each leave-one-out basket. The scores below are Epoch’s, from its score matrix. The GB300 multiples are our reciprocals of those scores, rounded.

The report itself quotes two H200 figures, which compare the full basket with Kimi K3 alone. It says that for Kimi K3 its model predicts “roughly a 46-fold GB300 advantage over H200, compared with eightfold across the full basket.” The full per-model range is wider: from about 2.7x on Qwen3-Coder to about 63x on the MAI proxy. Epoch’s study gives the same warning in general terms, saying a basket score “should not be presented as an architecture-independent hardware multiplier.”

Readers working from the study’s rounded tables should use the full-precision CSV for the extremes. The tables display H100’s Kimi score as 0.006, which on its own could mean anything from about 154x to 182x. The study also describes single-model results as “architecture diagnostics, not equally plausible replacement baskets.”

Why the arithmetic mean favors Hopper

The headline takes a simple average of four ratios. When one ratio is much larger than the others, it dominates. H100’s Qwen3-Coder ratio is 0.125, while its other three are between 0.005 and 0.023. By our arithmetic, Qwen3-Coder supplies about 79 percent of H100’s headline score, and about 75 percent of H200’s. The 25x headline is, in large part, a statement about one model on which Hopper does comparatively well.

Epoch addresses this in a footnote. “An arithmetic mean preserves a chip’s contribution on models it serves well,” it says, while a geometric mean “penalizes uneven performance more strongly, substantially lowering Hopper’s scores, but makes comparisons independent of the reference chip.” Epoch does not publish geometric-mean scores in the files we read. Computing one from its per-model scores gives about 0.017 for H100 and 0.057 for H200, which would put GB300’s advantage near 59x and 17x.

The reference point matters too. Because each ratio is taken against GB300 before averaging, the result depends on which chip is the reference. As an illustration, averaging the four ratios with H100 as the reference instead would score GB300 at about 107 H100 equivalents, not 25. Neither answer is wrong. They answer different questions, and the geometric mean is the only one of the three that gives the same ratio whichever chip is the reference.

Speed band and other choices

The model basket is not the only lever, but with one exception the others move H100’s comparison much less. Using Epoch’s published scenario scores:

The 400-token row is the exception. At that speed H100’s modeled capacity across the basket falls to 0.0037 GB300e, so GB300’s advantage jumps above 270x. H200 does not show the same cliff. The integrated headline smooths over such points, which is part of its purpose, but a buyer whose agents must generate very quickly should look at the single-speed scores, not the average.

For context on other chips, Epoch scores AMD’s MI355X at about 0.55 and Vera Rubin at 2.18, using a revised Rubin profile from 20 October that lowered Rubin’s score from 2.389. The frozen release covers 23 accelerator profiles, and the repository notes this is not an assertion that all 23 are shipping products.

Modeled, not measured

The report says plainly that its scores “describe modeled hardware potential” and “do not account for differences in software maturity.” The question is how far the model can be trusted, and the validation study answers that for a narrow case.

Epoch compared its modeled curves with SemiAnalysis InferenceX measurements retrieved on 19 October 2026. For Kimi K3 on GB300, with both prefill and decode set to 50 percent of modeled SOL, predicted throughput differed from measured throughput by 11.6 percent on average at five frontier points in the 50 to 400 band. The median gap was 11.1 percent and the largest was 23.5 percent. Epoch says the 50 percent assumption was specified before the calculation and was not fitted. The modeled winner at each speed could use a different submitted layout from the measured winner.

The same study shows how much the agreement depends on that setup:

Epoch calls these “descriptive service-envelope gaps,” not a ranking of simulator or vendor accuracy, because workloads, contexts, precision and serving policies differ. It names Kimi on GB300 in its short explanation “rather than generalizing its 12% agreement to all platforms.”

No Hopper row appears. The validation notes that “H200 Kimi has no observed point reaching the 50-TPS band” in the saved InferenceX data. That is consistent with the model’s very low Kimi score for Hopper, but it is not a measurement of the gap. SemiAnalysis’s 25 August AgentX post attributes Hopper’s weak Kimi K3 results partly to software. It says Hopper “requires custom tuned kernels for K3” and lacks tuned shapes for high-interactivity serving, concluding that “the hardware is not the binding constraint so much as the absence of anyone working on it.” Epoch’s report acknowledges this, noting that InferenceX “attributes part of Hopper’s weak Kimi K3 performance to missing software optimizations.”

The broader ordering does agree. Epoch reports that InferenceX’s Kimi K3 benchmarks rank Rubin and GB300 ahead of B300 and MI355X, with H200 far behind. That is evidence about order, not about the size of the multiples.

A reconstructed timeline

The report’s date and the dates of its inputs differ by weeks. The sequence below is reconstructed from the report, the repository’s snapshot guide and the cited sources.

The project’s earlier name, Inference Capacity Index, survives in some legacy identifiers. The repository says the rename changed no scores or methods.

Using GB300 equivalents for a fleet

Epoch proposes the GB300 equivalent as a common unit for a data center’s or even the world’s inference capacity. The arithmetic is simple, and the basket problem carries straight through it.

Take a hypothetical fleet of 10,000 H100s and 2,000 H200s. Under the headline basket it is worth about 397 plus 246, or roughly 640 GB300e. If the workload looks like Qwen3-Coder, the same fleet is worth about 1,990 GB300e. If it looks like Kimi K3, it is worth about 107. The hardware is identical in all three cases, and the estimate varies by a factor of almost 19.

The unit also differs from an earlier Epoch use of the same phrase. Epoch’s 2 October report, also by Jason Li, counted shipped high-bandwidth memory in units of 288 GB, matching a GB300. After assuming that newer HBM4-class systems support twice as many concurrent agents per unit, its appendix estimates about 60 million effective GB300 equivalents shipped through 2027. That is a memory-based equivalence with an assumed uplift for newer memory. The IPI’s GB300e is a modeled capacity ratio. A reader combining the two should not assume a chip counts the same under both.

Epoch suggests pairing the index with a model-specific estimate to get a rough count of agents served at once. The index supplies the hardware conversion, but the per-model number has to come from the workload actually being served.

What remains unknown

Several gaps limit what the 25x can support, and Epoch states most of them.

Software maturity. The scores describe potential under modeled limits. Production systems reach some fraction of that, and the fraction differs by chip, model and serving engine. Epoch expects AI-assisted optimization to narrow the gap over time, but that is a forecast. Hopper’s Kimi results are the clearest case where software, not only hardware, may explain measured weakness.

Precision and quality. Blackwell profiles use FP4 experts and Hopper profiles use FP8. Epoch writes that it has “not established that these choices preserve equivalent model quality.” The H100 catalog note refers to a separate weight-only FP4 sensitivity that “is not this score,” and we did not read its results.

One context length. Every request carries 262,144 cached tokens. The validation study notes that many observed contexts average less than that. Shorter contexts free memory for more requests, which should narrow gaps driven by capacity, but the index does not test it.

Missing serving behavior. The method excludes queueing, time to first token, KV transfer between prefill and decode pools, tool pauses and scheduling interference. It assumes fully utilized pools and fractional allocation at fleet scale.

Conservative layouts for small domains. Eight-GPU profiles use a serial pipeline schedule. A better-overlapped schedule could raise Hopper’s modeled capacity on the models that need many stages. The study does not report how much.

No error bars. The sensitivity ranges are, in Epoch’s words, “not confidence intervals or probabilities.” The index assigns no statistical uncertainty to its scores.

Evidence that would change the conclusion would include measured AgentX runs for H100 or H200 at long contexts with tuned kernels, a published FP4 weight sensitivity for Hopper, and an overlapped pipeline schedule in the search. As a proposed test, not one Epoch has announced, the cleanest check would replay one measured Hopper deployment at its observed batch, context and precision and compare it with the modeled point. The validation study already lists replaying measured winning deployments, and extending checks to Qwen and the MAI proxy, as the most useful next evidence.

What the comparison supports

The index makes one claim well. On long-context agent workloads, FLOP/s is a poor guide to serving capacity, because memory and interconnect set how many requests a system can hold and decode at a useful speed. The B300 versus GB300 result shows that clearly, with near-identical chips separated mainly by domain size.

The specific 25x is weaker evidence. It is a modeled average over four architectures. Most of H100’s score comes from the one model it handles best, the averaging rule is favorable to uneven chips, and no Hopper measurement checks it. A reader who needs a single number should use Epoch’s, cite the version and basket, and keep the per-model range next to it. For H100 that range runs from about 8x to about 217x, and the workload decides where in it a real fleet lands.

Epoch itself says hardware comparisons “should use scores from the same IPI version, since changing the basket can change a chip’s score even when its hardware is unchanged.” That is the right way to read the result. Our account of why reasoning models raise per-request costs shows the same pattern on the demand side, where the workload sets the price.