Thinking Budgets Are a Product Decision: Tuning Reasoning Effort per Task, Not per Model

A single parameter in your API call can triple your inference bill or triple your error rate, and most teams have never run an experiment on it. They picked reasoning_effort: "medium" during the initial build, or accepted the default thinking budget, and that value has shipped with every request since - the trivial ones, the hard ones, the ones where the answer routes a customer refund and the ones where it picks a label from a list of six.

That was defensible in 2024, when reasoning was a model property. You chose a reasoning model or you didn't. It stopped being defensible once every major provider turned thinking into a per-request dial: OpenAI's reasoning models accept reasoning_effort from minimal to high, Anthropic's extended thinking takes an explicit budget_tokens value, and Gemini exposes a thinkingBudget you can set anywhere from zero to tens of thousands of tokens. The dial exists on every request you send. Leaving it at one global value is a decision too, just an unexamined one.

The Dial Is Per Request, Your Config Is Per Deploy

Look at how the three big APIs shape this parameter and a pattern jumps out.

All three made it a request-level field, not an account setting or a model variant. The providers are telling you how they expect it to be used. Meanwhile, in most codebases we review, the value lives in an environment variable or a model-config object - one value per model, set at deploy time, shared by every call site.

The mismatch has a real price. Thinking tokens are invisible to your users but fully visible to your invoice. A request that spends 8,000 tokens reasoning and 400 tokens answering is, economically, a request that produced 8,400 output tokens. Artificial Analysis has documented reasoning models consuming several times the output tokens of non-reasoning models on identical evaluation suites, which is exactly what you'd expect when the thinking is on for every prompt regardless of difficulty. If you already do model routing in production, you've accepted that different requests deserve different models. Effort tuning is the same argument one level down, and it's often the cheaper lever, because you keep one model, one prompt, one set of learned failure modes, and vary only the budget.

The Quality Curve Flattens Before the Price Curve Does

If accuracy scaled linearly with thinking tokens, one global high setting would be fine and this article would be pointless. It doesn't. Three separate lines of evidence say the returns diminish fast.

First, published effort-vs-accuracy benchmarks. Cost-versus-quality comparisons across effort levels consistently show the medium setting landing within a few points of high on general reasoning suites while consuming a fraction of the tokens. The gap between low and medium is usually larger than the gap between medium and high. You pay full price for the last notch and get the smallest increment of quality from it.

Second, the overthinking research. A 2024 study bluntly titled "Do NOT Think That Much for 2+3=?" measured reasoning models spending thousands of tokens on arithmetic a small model answers instantly, generating multiple redundant solution paths for problems they had already solved correctly in the first hundred tokens. The extra thinking wasn't checking anything. It was expensive idling.

Third, and most interesting, the failure end of the curve. Apple's Illusion of Thinking paper found that on controllable puzzle tasks, reasoning traces grow longest right in the difficulty band where accuracy is collapsing, and then, counterintuitively, models start thinking less as problems get harder still, as if giving up. More budget did not rescue performance past the collapse point. There is a region where extra tokens buy you nothing but latency.

Put those together and the shape of the curve is clear. There's a steep early section where thinking budget genuinely converts to accuracy, a flat middle where you're paying for redundant verification, and a far end where no budget helps. The Stanford s1 test-time scaling work showed the early section is real and controllable - forcing a model to think longer with as little as a budget cue measurably improved math performance - so the dial works. The engineering question is where your task sits on that curve, and you can't answer it with one global setting.

Bind Effort to Step Class, Not to the Model

Here's the framework we use. Agent workflows decompose into a small number of step classes, and each class has a characteristic position on the effort curve. Set budgets per class, and every new workflow inherits sensible defaults.

Classification and routing. Picking an intent label, choosing a tool, deciding which branch of a workflow fires. These are usually answerable from surface features of the input. Run them at minimal effort or thinking disabled. If a routing step needs 4,000 thinking tokens to pick between five branches, the fix is a better-specified routing prompt or clearer branch definitions, not a bigger budget.

Extraction and transformation. Pulling fields from a document, reformatting, mapping between schemas. Low effort. Schema validation on the output catches the failures more cheaply than thinking prevents them.

Planning and decomposition. Turning a goal into a sequence of steps, deciding what to do about an ambiguous instruction, choosing among approaches with different tradeoffs. This is the steep section of the curve. Medium to high effort, and worth benchmarking at the top settings, because a bad plan poisons every downstream step and the downstream retries cost more than the upfront thinking.

Verification and repair. Diagnosing why a tool call failed, reconciling contradictory retrievals, fixing code that didn't pass tests. High effort, and this is the one class where escalation beats a static setting - first attempt at medium, retry at high. The retry-with-more-thinking pattern converts the effort dial into a fallback tier, the same way you'd fall back to a bigger model, except cheaper because nothing else about the call changes.

Generation for humans. Drafting the email, writing the summary. Mostly low effort. Fluency doesn't come from thinking tokens, and a human reviews the output anyway.

Two consequences of arranging it this way. The obvious one: a workflow with ten classification steps and one planning step stops paying planning prices ten times per run. We walked through the arithmetic of steps-times-tokens-times-runs in our agent cost engineering piece, and thinking tokens enter that multiplication with full weight. The less obvious one is latency. Thinking happens before the first visible token, so effort settings on user-facing steps set your time-to-first-token floor. A support agent that runs its routing step at high effort adds seconds of dead air before anything happens, for a decision that minimal effort gets right at nearly the same rate.

Latency is also why "just set everything to high, tokens are getting cheaper" fails as a strategy. Token prices fall; the seconds a user waits do not. A 20-second thinking pause on an interactive step is a product defect at any price.

Put the Dial in the Eval Matrix

Step-class defaults get you to a sane starting point. The stop-gain point - the budget where your accuracy curve for your task goes flat - is something you measure, and the measurement belongs in the same harness as your other regression checks. We've argued before that generic benchmarks don't predict workflow performance, and effort tuning is a clean special case: the published curves tell you the shape to expect, not where your task's knee sits.

The procedure is short:

1. Take the eval set you already have for a workflow step (if you don't have one, that's the prerequisite, not this).
2. Run it at three budget levels - roughly minimal, medium, and high, or 1k / 8k / 24k thinking tokens on APIs that take explicit counts.
3. Plot accuracy against total cost per request and against p95 latency.
4. Set the production budget one notch above where accuracy flattens. The notch is margin for input drift.
5. Record the setting next to the model ID and prompt hash, and re-run the sweep whenever any of the three changes.

Step 5 is the one teams skip. Effort settings interact with model versions in unpredictable ways - a model upgrade can shift the knee of the curve in either direction, because newer models often need less thinking for the same result. A budget tuned for last quarter's model silently overpays on this quarter's. Treat the sweep as part of the migration checklist, the same as prompt regression tests.

One warning from doing this on real workloads: watch accuracy at high effort, don't assume it's a safe ceiling. On tasks with a verifiable answer, overthinking mostly wastes money. On tasks with judgment calls, long reasoning traces sometimes talk the model out of a correct first instinct, and you'll see the curve bend slightly downward at the top setting. That's not a benchmark artifact; it shows up in production traces. When it does, the cheap setting is also the correct one, which is a pleasant thing to be able to prove to whoever reviews the bill.

Making Effort Tuning Part of the Build, Not an Afterthought

For readers mapping this onto a buy-versus-build decision, the dial cuts both ways. If you're on a platform product that wraps the model call, check whether it exposes per-step reasoning control at all; many automation platforms pin one setting per connection, which locks you into exactly the global-default problem this article is about. If it does expose the dial, platform plus your own eval sweep is a fine path, and you don't need a consultancy for it. Custom builds earn their keep when the workflow has enough step diversity that per-class budgets and retry escalation matter, which in our experience starts around the point where one workflow mixes interactive steps with heavy planning or repair steps.

When OpenNash builds an agent system, effort budgets are set during the design phase alongside guardrails and approval points, one budget per step class, with the sweep from the previous section wired into the eval harness the client keeps after handoff. The deliverable includes the curve data, so when a model migration or a pricing change lands, whoever owns the system re-runs the sweep and re-derives the settings instead of inheriting a magic number. That ownership matters more here than in most tuning decisions, because the budgets go stale on every model release.

If you want to gauge the size of the opportunity before committing to anything, the diagnostic is one query against your own logs. Pull a week of traces, group by workflow step, and sum thinking or reasoning tokens per group. Most teams find one or two steps consuming the large majority of thinking spend, and at least one of them is a step class that shouldn't be thinking much at all. Run the three-level sweep on your top spender this week; it's an afternoon of work against an eval set you should already have, and it usually pays for itself before the month is out.