Spending Compute at Inference · Advanced Reasoning, Planning & Self-Improving Agents

The quarterly eval report is on the screen and the number is 62%. That is the share of support tickets your agent resolves without a human touching them, measured on a frozen golden set, with a confidence interval you trust because you built the harness yourself. Product wants 70% by the end of the quarter. The obvious move is the next model tier up: roughly five times the price per token, a rebuild of every prompt you have tuned, and a two-week migration. Then the engineer across the table asks the question that reorganizes the whole meeting: what if we just let the current model think longer, or answer eight times and keep the best one?

That question is the subject of this course. For most of the deep learning era, the way you got a better answer was to train a better model — more parameters, more data, more pretraining FLOPs — and inference was a fixed, cheap afterthought. Since 2024 that has stopped being the only lever. A model that spends more compute at inference — by producing a longer reasoning trace, or by producing many candidate answers and selecting among them — can behave like a substantially larger model on the tasks you care about, at a cost you control per request rather than per training run. Reasoning models made the sequential version of this the default product behavior; verifier-guided sampling made the parallel version a standard engineering pattern.

This lesson gives you the map: the two axes along which inference compute can be spent, what the compute-optimal scaling literature actually shows (and, more importantly, what it does not), why sampling helps enormously on some problems and not at all on others, and how to run the cost arithmetic before you commit. By the end you should be able to walk into that meeting and say which axis you are buying, how many dollars per solved task it costs, and where the curve stops paying.

1. Two Axes, One Budget

Every technique in this section converts inference FLOPs into accuracy, and there are exactly two directions to spend in.

The sequential axis makes one attempt longer. The model emits more tokens before committing to an answer: it works through cases, checks its arithmetic, notices a contradiction and revises. That is what a reasoning model does when it produces a long chain of thought before the visible response. The knob is a thinking budget — the maximum number of reasoning tokens the model may spend — and every major model family in 2026 exposes some version of it: extended-thinking token budgets on Claude, reasoning effort levels on OpenAI's o-series, thinking budgets on Gemini, configurable generation length on open-weight reasoners like DeepSeek-R1 and the Qwen reasoning line.

The parallel axis makes many independent attempts. Sample $n$ complete answers at nonzero temperature, then pick one: by majority vote if the answers are discrete, by a learned verifier's score, or by running the candidate code against tests. This is best-of-n, and it is old — Cobbe et al. used exactly this structure with a trained verifier on GSM8K in 2021 — but it has become a first-class scaling knob because inference stacks like vLLM and SGLang make $n$ samples that share a prompt prefix cost far less than $n$ times one sample.

The axes are not interchangeable, and the difference that matters most in production is not accuracy, it is latency. Parallel compute is wall-clock free if you can fan out: eight samples of 1,200 tokens each finish in roughly the time of one, because they generate concurrently. Sequential compute is not: 9,000 reasoning tokens is 9,000 tokens of serial decode, and at a realistic 60 tokens per second that is two and a half minutes the user is watching a spinner. If you have an interactive SLA, the parallel axis is usually the only one available to you.

2. Why Inference Became a Scaling Frontier

Three things happened at once. The first is economic: a frontier pretraining run is a capital decision measured in months and hundreds of millions of dollars, and it improves every task at once by a bit. Test-time compute is an operating decision measured in cents per request, and you can aim it at exactly the 8% of your traffic that is hard.

The second is that reinforcement learning with verifiable rewards taught models to use a long context productively. Before 2024, telling a model to think longer mostly produced longer restatements of the same wrong answer. The RLVR recipe — reward the final answer against a checker, let the policy discover whatever reasoning gets it there, as in DeepSeek-R1's GRPO training — produced models whose accuracy genuinely keeps climbing with trace length, because the trace now contains backtracking, case analysis, and self-checking rather than filler. That training story is covered in depth in Model Specialization: Fine-Tuning & Agentic RL; here you are on the consuming end.

The third is that serving infrastructure made parallel sampling cheap. When eight completions share one 4,000-token prompt, a modern engine prefills that prompt once and shares the KV cache across all eight sequences, while continuous batching keeps the GPU saturated. Best-of-8 ends up costing closer to 5x a single call than 8x — a difference that decides whether the technique is viable at all.

3. Compute-Optimal Scaling, and Its Fine Print

The reference result is Snell, Lee, Xu and Kumar (2024), Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters — the paper people cite when they claim a small model plus search beats a big model, so be precise about what it established. The authors studied two mechanisms — verifier-guided search against a process reward model, and sequential revision, where a model trained to improve its own answers iterates on a draft — and asked how to allocate a fixed inference budget between them. Their central finding is that the best allocation depends strongly on question difficulty, and that a policy which estimates difficulty per prompt and adapts the strategy accordingly is several times more efficient than any single fixed strategy such as plain best-of-n. Then, in a FLOPs-matched comparison, they showed that on easier and mid-difficulty questions, spending the budget on test-time compute with a smaller model outperformed a roughly 14x larger model doing single-shot inference.

Now the fine print, which is where engineers get hurt:

It flips on the hardest questions. On the top difficulty bins, the larger pretrained model won. Test-time compute amplifies capability the base model already has; it does not create capability the base model lacks.
It flips at large inference budgets. The advantage of search holds when the inference budget is modest relative to pretraining compute. Push the per-question budget far enough and you are better off having trained the bigger model.
It depends on a good verifier. The search results lean on a process reward model trained for that domain. Without a reliable scorer, the search has nothing to steer by — which is the subject of Lesson 3.
The domain is math. The experiments are on MATH, a domain with unambiguous discrete answers and dense step structure. Your ticket-triage agent is not that.

4. Coverage Versus Selection

Two separate questions hide inside "did sampling help", and keeping them apart is the single most useful analytical habit in this course. Coverage asks: does at least one of my $n$ samples contain the right answer? If each independent sample succeeds with probability $p$ , coverage is

coverage (n) = 1 - (1 - p)^{n}

which is the familiar pass@k quantity from Evaluation & Benchmarking of Agentic Systems. Brown et al. (2024), Large Language Monkeys, showed that coverage keeps rising remarkably far in practice — often close to log-linearly in the number of samples across several orders of magnitude, on coding and math benchmarks. Sample a weak model enough times and it eventually emits a correct solution to problems it "cannot solve".

Selection asks the harder question: can you tell which of the $n$ it was? This is where the story splits. In domains with an automatic checker — code with a test suite, a theorem with a proof checker, a SQL query you can execute against a fixture — selection is free and correct, so coverage is your accuracy, and repeated sampling is spectacular. In domains without one, you fall back on majority voting or a learned reward model, and Brown et al. found those methods plateau well before coverage does, typically somewhere in the low hundreds of samples. Coverage climbing while selected accuracy sits flat is the signature failure mode of the parallel axis, and it means your problem is a verifier problem, not a sampling problem.

There is a third case that no amount of either axis fixes. If $p$ is effectively zero — the task requires a tool the agent does not have, a fact absent from its context, or a reasoning step the model has never been able to make — then coverage is zero for any $n$ , and a longer thinking budget produces a longer confident wrong answer. Multiplying zero is still zero. Before you buy compute, look at the failure transcripts and confirm the model gets it right sometimes. If it never does, you need capability: a tool, a retrieval fix, a fine-tune, or a different model.

Your code agent solves 41% of tasks single-shot. You sample 16 candidates per task and find that 89% of tasks now have at least one passing candidate, but the answer you actually return is right only 52% of the time. What is the highest-value next move?

5. Sequential Knobs: Thinking Budgets and Budget Forcing

On the sequential axis your primary control is a token budget, and the interesting question is what happens at the boundary. Muennighoff et al. (2025), s1: Simple test-time scaling, gave the cleanest demonstration with a technique they call budget forcing: to make the model think less, append the end-of-thinking delimiter and force it to answer now; to make it think more, suppress that delimiter and append a continuation token — literally the word "Wait" — so decoding continues. The model, finding itself still inside its reasoning block, frequently re-examines and corrects its own work. With a 1,000-example supervised set and this decoding-time trick, their model's accuracy climbed monotonically as the forced budget grew.

Two operational lessons come out of that. The useful range of a thinking budget is bounded — extend far enough and you get repetition loops and eventually degradation. And budgets must be enforced by your harness, not requested politely in the prompt: "think briefly" is not a budget, a hard cap on reasoning tokens with a forced answer at the cap is. That is the loop-level cost control discussed in Observability, Reliability & Cost Engineering, and a reasoning model without one is an open-ended invoice.

A third knob sits between the axes: revision, where you feed a complete answer back with a critique and ask for a better one. Each round starts from a concrete artifact rather than a partial thought, which makes it behave differently from one long chain — but it only pays when the critique comes from outside the model: a test run, a type checker, a retrieved fact. Self-critique with no external signal mostly ratifies whatever the model already said, a result Lesson 3 returns to with the evidence.

6. The Cost Arithmetic You Owe Your Finance Team

Here is the calculation to run before you reach for either axis. Take a realistic agent request: a 4,000-token prompt and a 1,200-token answer, on a mid-tier 2026 model priced around 3 dollars per million input tokens, 15 per million output, with cached prompt reads at roughly a tenth of the input rate.

Strategy	Tokens billed	Cost / request	Wall clock at 60 tok/s
Single shot	4k in, 1.2k out	about 3.0 cents	about 20 s
Best-of-8, shared prefix	4k in + 7x cached, 9.6k out	about 16 cents	about 22 s (fanned out)
Thinking budget 8k	4k in, 9.2k out	about 15 cents	about 155 s (serial)
Best-of-8 + small verifier	above + 8 x 5.2k verifier in	about 17 cents	about 24 s

Two things jump out. Best-of-8 and an 8,000-token thinking budget cost about the same money and buy roughly comparable accuracy gains on many tasks — but one of them is seven times slower in wall clock. And a verifier built on a small model is nearly free relative to generation. Verification is almost always the cheapest part of the system, which is why the rest of this section works so hard at making it good.

Now convert accuracy into money. Suppose best-of-8 moves you from 62% to 75% resolution, and each unresolved ticket costs about 12 minutes of a support agent's time — call it 16 dollars fully loaded. The marginal spend is roughly 13 cents per ticket; the expected saving is 0.13 x 16, or about 2 dollars. That is a 15x return and an easy decision. Run the same arithmetic where a failure costs nothing but a retry, and the same 13 cents is indefensible. The value of a marginal correct answer is the number that decides this, and it is a business number, not an ML one — which is why the metric on your dashboard should be cost per successfully completed task, not accuracy per token.

7. A Decision Rule You Can Defend

Put the pieces together into something you can apply in a design review. Work down this list; the first row that matches tells you what to do.

What you observe	What to spend on
Model is never right on these tasks, at any temperature	Neither axis. Fix capability: tools, context, retrieval, or a specialized model.
Coverage at n=8 is far above selected accuracy	Selection. Build or buy a verifier before buying more samples.
Answers are discrete and checkable; latency budget is tight	Parallel. Best-of-n with an automatic checker, fanned out.
Errors are local slips the model can catch on re-reading	Sequential. Larger thinking budget, or a revision round with an external signal.
Errors are wrong-approach failures needing a different plan	Parallel breadth first, then search over the branches (Lesson 5).
Marginal dollars per extra solve exceed the value of a solve	Stop. You are past the compute-optimal point for your economics.

You can now do three things you could not before: name which axis a proposed change is buying and what it does to latency, separate a coverage problem from a selection problem using numbers you can measure in an afternoon, and state the break-even point of a compute spend in dollars rather than adjectives. The next lesson turns the parallel axis into working code and makes you plot the curve where it flattens.