Test-time compute, explained: the second scaling lever

TL;DR

• Test-time compute is computation spent while a model answers, not while it is trained. Scaling it means longer reasoning, more samples, verifiers, search, revision, or a model trained to do all of that on its own.
• Snell et al. (2024) found that a compute-optimal allocation beats a best-of-N baseline by more than 4x and, in a FLOPs-matched comparison, can outperform a 14x larger model, but only on problems the small model can already sometimes solve.
• Reasoning models such as OpenAI’s o1 and DeepSeek-R1 moved the idea from prompting tricks into training: reinforcement learning teaches the model to produce a long chain of thought at inference.
• The limits are real. Gains depend on verifiable answers, selection cannot find a correct answer that was never sampled, and every reasoning token is billed as output.

For most of the last decade the recipe for a better language model was fixed: more parameters, more data, more training compute. Test-time compute (also called inference compute or inference-time scaling) is the name for a second lever. A trained model is a fixed function: given a prompt it produces one answer in a single pass, and until recently that was how models were used. Test-time compute means spending more than that single pass, so a fixed cost per query becomes a variable one that can be matched to how hard the question is. This explainer walks through the research that established the lever, from chain-of-thought prompting in 2022 to reinforcement-learned reasoning models in 2025, states the headline numbers as the papers state them, and sets out where the approach stops working.

Chain of thought and self-consistency

The first widely adopted form of test-time compute was simply letting the model write more. Wei et al. (2022) showed that providing “a few chain of thought demonstrations as exemplars in prompting” makes large models write out intermediate reasoning steps, which “significantly improves the ability of large language models to perform complex reasoning.” Their headline example: prompting a 540B-parameter model with “just eight chain of thought exemplars” reached state-of-the-art accuracy on GSM8K, a benchmark of grade-school maths word problems, “surpassing even finetuned GPT-3 with a verifier.” Longer output is more compute per answer, and the paper found the benefit appears only in “sufficiently large” models.

The second form was parallel sampling. Wang et al. (2022) proposed self-consistency, which “replaces the naive greedy decoding used in chain-of-thought prompting.” The model samples several reasoning paths and the method “selects the most consistent answer by marginalizing out the sampled reasoning paths.” In plain terms: generate many answers, take a vote. The paper reports gains of 17.9 percentage points on GSM8K, 11.0 on SVAMP, 12.2 on AQuA, 6.4 on StrategyQA and 3.9 on ARC-challenge.

Voting has a weakness. It picks the most common answer, not the best one. If the model’s usual mistake is also its most common answer, voting confirms the mistake.

Verifiers and process reward models

The fix is a second model that judges the candidates. Lightman et al. (2023), “Let’s Verify Step by Step,” compared two ways of training such a judge: outcome supervision, “which provides feedback for a final result,” and process supervision, “which provides feedback for each intermediate reasoning step.” The process-supervised judge is a process reward model, or PRM. The paper finds that “process supervision significantly outperforms outcome supervision” on the MATH dataset of competition problems.

At test time the PRM is used for best-of-N selection. The authors “select the solution ranked highest by the reward model,” where a solution’s score is “the probability that every step is correct under the PRM,” implemented “as the product of the correctness probabilities for each step.” Picking among samples this way, the model solved “78.2% of problems from a representative subset of the MATH test set.” The PRM’s advantage over the outcome model and majority voting grows as N increases. The paper also released PRM800K, a dataset of 800,000 step-level human labels.

Repeated sampling and coverage

How far can pure sampling go? Brown et al. (2024), “Large Language Monkeys,” measured coverage, “the fraction of problems that are solved by any generated sample,” and found it “scales with the number of samples over four orders of magnitude.” The relationship “is often log-linear and can be modelled with an exponentiated power law,” which the authors read as evidence for “inference-time scaling laws.”

The result is striking where answers can be checked automatically. On SWE-bench Lite, a set of real GitHub issues with tests, the fraction solved by DeepSeek-Coder-V2-Instruct “increases from 15.9% with one sample to 56% with 250 samples, outperforming the single-sample state-of-the-art of 43%.” The paper adds that “sampling five times from the weaker and cheaper DeepSeek model solves more issues than single samples from Claude or GPT while also being over 3x cheaper.”

The same paper reports the ceiling. Without an automatic verifier, “majority voting and reward models” plateau “beyond several hundred samples and fail to fully scale with the sample budget.” On maths tasks the authors report coverage of 98.44% with 10,000 samples while selection methods reached 41.41%. The correct answer was in the pile; nothing could reliably pick it out.

Compute-optimal scaling and the 14x claim

The paper most often cited for “test-time compute can replace model size” is Snell et al. (2024), “Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters.” Working with PaLM 2-S on the MATH benchmark, the authors studied two mechanisms: “searching against dense, process-based verifier reward models” and “updating the model’s distribution over a response adaptively,” which in practice means sequential revision of a draft. Their central finding is that “the effectiveness of different approaches to scaling test-time compute critically varies depending on the difficulty of the prompt.” Easy questions benefit from “purely sequential test-time compute” (revisions); harder ones do best “with some ideal ratio of sequential to parallel compute.” Beam search over a PRM “degrades performance” on easy questions but “consistently outperforms best-of-N” on harder ones. Difficulty was estimated by binning each question’s pass@1 rate into five quantiles.

That observation motivates the “compute-optimal” strategy: choose the best test-time method per difficulty level rather than one method for everything. With it, the paper reports it “can improve the efficiency of test-time compute scaling by more than 4x compared to a best-of-N baseline.”

The 14x claim is narrower than it is usually quoted. The exact wording is: “in a FLOPs-matched evaluation, we find that on problems where a smaller base model attains somewhat non-trivial success rates, test-time compute can be used to outperform a 14x larger model.” Section 7 spells out the other side. On “the most challenging questions” the authors “observe very little benefits from scaling up test-time compute” and find “it is more effective to make progress by applying additional pretraining compute.” Their summary: “test-time and pretraining compute are not 1-to-1 ‘exchangeable’.”

Reasoning models trained with reinforcement learning

Everything above applies test-time compute from outside the model: prompts, samplers, verifiers, search loops. The next step was to train the behaviour in.

OpenAI’s o1-preview arrived on 12 September 2024. The company’s post, “Learning to reason with LLMs,” states that “performance of o1 consistently improves with more reinforcement learning (train-time compute) and with more time spent thinking (test-time compute),” a claim illustrated with a scaling chart for each axis. The o1 system card describes the series as “trained with large-scale reinforcement learning to reason using chain of thought.” The reasoning is hidden: OpenAI wrote that it did “not want to make an unaligned chain of thought directly visible to users,” and as Simon Willison noted at launch, “you get billed for them, but you don’t get to see what they were.”

DeepSeek-R1, published on 22 January 2025, showed the recipe in the open. DeepSeek-R1-Zero was “trained via large-scale reinforcement learning (RL) without supervised fine-tuning (SFT) as a preliminary step” and “naturally emerges with numerous powerful and intriguing reasoning behaviors,” though with “poor readability, and language mixing.” DeepSeek-R1 adds “multi-stage training and cold-start data before RL” and, the paper claims, “achieves performance comparable to OpenAI-o1-1217 on reasoning tasks.” The lab released both models plus “six dense models (1.5B, 7B, 8B, 14B, 32B, 70B) distilled from DeepSeek-R1 based on Qwen and Llama,” with weights “licensed under the MIT License” according to the repository.

Muennighoff et al. (2025) asked how simple the recipe could get. Their s1 paper fine-tuned Qwen2.5-32B-Instruct on s1K, “1,000 questions paired with reasoning traces,” and added budget forcing: “forcefully terminating the model’s thinking process or lengthening it by appending ‘Wait’ multiple times to the model’s generation when it tries to end.” The authors report that s1-32B “exceeds o1-preview on competition math questions by up to 27% (MATH and AIME24),” and that forcing longer thinking lifted AIME24 accuracy “from 50% to 57%.” A single knob, thinking length, moved the score.

Test-time compute did not replace scale. It gave every request a second dial, and the papers are clear about where that dial stops turning.

A taxonomy of test-time compute methods

The limits

Three constraints recur across these papers.

It works where answers can be checked. Brown et al. observe that coverage gains “directly translate into improved performance” in “domains like coding and formal proofs, where answers can be automatically verified.” DeepSeek-R1’s reported gains are on “verifiable tasks such as mathematics, coding competitions, and STEM fields.” For open-ended writing and questions with no ground truth, there is nothing for a verifier, a vote or a reward signal to grip, and the plateau Brown et al. measured is the expected outcome.

Selection cannot find what was never sampled. Coverage is the ceiling: if no sample contains a correct answer, no verifier will produce one. Snell et al. put the matching limit on the training side: on the hardest problems, where the base model’s success rate is near zero, pretraining compute did more than any test-time method. Thinking longer sharpens what a model can already almost do.

The bill scales with the thinking. Anthropic’s thinking documentation states that “the tokens Claude spends reasoning are billed as output tokens, even when the thinking text isn’t returned to you.” OpenAI’s reasoning guide says reasoning tokens “are not visible via the API” but “are billed as output tokens.” Our news desk covers what that does to unit economics in the reasoning-model era has a cost problem.

Thinking budgets in 2026 provider APIs

The research knobs have become API parameters, and by late 2026 the trend is from fixed budgets to adaptive ones.

Anthropic’s original control was a fixed budget, thinking: {type: "enabled", budget_tokens: N}, with a minimum of 1,024 tokens. The extended thinking page now marks that mode deprecated on the Claude 4.6 models and says Claude 4.7 and later reject it. The replacement is adaptive thinking, thinking: {type: "adaptive"}, steered by an effort level: “With a fixed budget, Claude thinks on every request. With adaptive thinking, Claude decides whether and how much to think on each request, and at lower effort settings it may skip thinking entirely on easy inputs.” The effort page lists low, medium, high, xhigh and max, and notes that effort “is a behavioral signal, not a strict token budget.”

OpenAI’s reasoning_effort parameter takes values the documentation lists as none, minimal, low, medium, high, xhigh and max, with availability model-dependent.

This is Snell et al.’s compute-optimal idea in product form: allocate thinking per prompt by difficulty rather than uniformly. The difference is who decides. In the paper, a validation set chose the strategy per difficulty bin; in adaptive thinking, the model decides on each request, and the buyer’s control is a coarse effort level.

What to take from it

Test-time compute is a measured lever, and the evidence is consistent across independent groups. It is also bounded in ways the same evidence makes explicit: the strongest gains come from verifiable tasks, from problems the base model can sometimes solve, and from allocating compute by difficulty.

The operational questions follow. Which requests deserve high effort, how to route between a reasoning model and a cheaper one, and how to see reasoning-token spend per request are decisions that sit in front of the model rather than inside it; our field notes on running an AI gateway in front of every model call cover that control point. And because the right amount of thinking depends on the task, the only way to know where your own dial should sit is to evaluate on your own traffic at each effort level, with the bill next to the accuracy.