A support agent that reads a 30,000 token case history and writes a 60 token answer is not a chatbot. It just looks like one on the invoice.
That single request does roughly 4 petaFLOPs of matrix math to read the input, then spends most of its wall clock time generating sixty tokens that barely touch the compute units at all. Two completely different machines are being asked to run on one piece of silicon. If your serving stack was configured when your traffic looked like chat, you are paying for that mismatch on every request, and the bill does not tell you which half is at fault.
Cost per token is not a price you shop for. It is an outcome of how you arranged the serving architecture around the shape of your traffic.
Two Workloads Wearing the Same Costume
An LLM request has two phases with opposite resource profiles.
Prefill processes the entire input prompt in one shot. Every token attends to every prior token, the math is dense matrix multiplication, and the GPU's tensor cores run hot. Prefill is compute bound.
Decode generates one token at a time. Each step reads the full model weights plus the KV cache out of high bandwidth memory, does a small amount of arithmetic, and writes one token. Decode is memory bandwidth bound.
The gap is not small. An H100 delivers roughly 990 TFLOPS of dense BF16 compute against about 3.35 TB/s of memory bandwidth, a ratio near 295 FLOPs per byte moved. Single-sequence decode has an arithmetic intensity closer to 2. Kipply's transformer inference arithmetic walks through the math, and the conclusion is uncomfortable: at batch size 1, decode uses well under one percent of the compute you are renting. You need a few hundred concurrent sequences before decode stops being bandwidth starved.
Run some numbers on a realistic agent step. A 70B model in BF16 is 140 GB of weights. Prefilling 30,000 tokens costs about 4.2 petaFLOPs, which on an eight-GPU node at a realistic 40 percent utilization takes somewhere near 1.3 seconds. Generating 80 output tokens at 40 tokens per second takes 2 seconds. Comparable wall clock, opposite bottlenecks. One phase wants FLOPs and the other wants bytes per second, and they are fighting over the same card.
Colocating them creates a second problem that is worse than the waste: interference. When a 30,000 token prefill lands on a GPU that is mid-decode for twelve other users, those users watch their tokens stall. This is the source of most inter-token latency spikes that get misdiagnosed as "the model is slow."
Your Token Shape Determines Everything
Before you evaluate a single serving feature, measure three numbers from your own traffic:
- P: median input tokens per request
- D: median output tokens per request
- R: fraction of input tokens that repeat across requests (shared system prompts, tool schemas, conversation history)
Take medians, not means. A handful of enormous context requests will drag an average somewhere useless.
Here are shapes we see across deployments:
| Workload | P (input) | D (output) | Ratio | Typical reuse (R) |
|---|---|---|---|---|
| Chat turn | 600 | 400 | 1.5:1 | 20-40% |
| RAG answer | 4,000 | 500 | 8:1 | 30-50% |
| Document extraction | 12,000 | 800 | 15:1 | 10-20% |
| Coding agent step | 25,000 | 150 | 165:1 | 70-90% |
| Tool-calling agent step | 18,000 | 80 | 225:1 | 80-95% |
The jump from row two to row five is the whole story. Chat-era serving defaults assume prefill is a small tax on a long generation. Agent loops invert that: they read enormous context and emit a short tool call, then do it again. A ten-step agent run does ten prefills and ten tiny decodes.
Most teams that tell us their inference costs "came in higher than the pricing page implied" benchmarked with chat-shaped prompts and deployed an agent-shaped workload. The per-token price never changed. The token shape did, and the serving configuration never followed.
This is the same failure pattern we described in the agentic AI memory bottleneck, viewed from the hardware side instead of the application side.
Lever One: Split Prefill From Decode
Disaggregation means running prefill and decode on separate GPU pools and shipping the KV cache between them over fast interconnect.
The idea got its rigorous treatment in two 2024 systems papers. Microsoft's Splitwise split the phases across machine pools and reported roughly 1.4x higher throughput at 20 percent lower cost, or about 2.35x more throughput under the same power budget, partly because prefill and decode pools can run on different hardware generations. DistServe attacked it from the goodput angle, optimizing time-to-first-token and time-per-output-token as separate service levels, and reported serving several times more requests within the same latency constraints on their benchmarks.
What disaggregation buys you, concretely:
- Independent scaling. If your ratio is 200:1, you need far more prefill capacity than decode capacity. Colocated serving forces those into a fixed ratio determined by whatever GPUs you happened to buy.
- Independent hardware. Decode pools want memory bandwidth and can run on older, cheaper accelerators. Prefill pools want raw FLOPs. Buying one SKU for both means overpaying on one side.
- No interference. A long prefill cannot stall someone else's token stream because it is not on that machine.
- Independent parallelism. Prefill often wants tensor parallelism for latency. Decode often wants larger batches and different sharding. Disaggregation lets you set both.
What it costs you: KV cache transfer bandwidth, a more complicated control plane, and a real risk of stranded capacity if your traffic mix shifts. The cache for a 30,000 token context on a 70B model runs to several gigabytes, so you need serious interconnect between pools. Below roughly a few nodes of scale, the operational overhead is not worth it, and the accounting mistake we see most often is treating idle prefill capacity during off-peak hours as free. It is not. The scaling tradeoffs here connect directly to the four ways to scale AI compute.
Both vLLM and SGLang now ship disaggregated modes, and NVIDIA's Dynamo exists specifically to orchestrate prefill and decode pools across a cluster. The feature is available. Whether you should turn it on depends entirely on the numbers you measured in the previous section.
Lever Two: Fix the Scheduler Before You Buy More GPUs
Before disaggregating, exhaust the scheduling wins. They are cheaper and they compound.
Continuous batching replaced static batching around 2022 with the Orca paper's iteration-level scheduling: finished sequences leave the batch immediately and new ones join at the next step, instead of everyone waiting for the slowest request. If you are not running this, nothing else matters yet.
Chunked prefill splits a long prompt into pieces and interleaves them with ongoing decode steps. Sarathi-Serve formalized this as stall-free batching and reported serving capacity improvements in the 2.6x to 5.6x range depending on model and hardware. It is the poor team's disaggregation: you keep one pool, but you stop letting long prefills block token streams. For many mid-sized deployments, chunked prefill captures most of the interference benefit of disaggregation at a fraction of the operational cost.
Prefix caching is the lever that is not on the list and should be first. Agent loops resend the same system prompt, the same tool schemas, and a growing conversation history on every single step. That is what the R column measures. If 85 percent of your input tokens are a repeated prefix, caching the KV state for that prefix removes 85 percent of your prefill compute. SGLang's RadixAttention generalized this to a shared tree across requests, so overlapping prefixes across different users hit the same cache.
The numbers here get absurd in a good way. Character.AI's engineering team published their inference optimization work showing a 95 percent cache rate and a claimed 33x reduction in serving cost since 2022, built on aggressive KV cache reduction and cross-request cache reuse rather than exotic hardware.
Prefix caching is a configuration change and a prompt-ordering discipline: put the stable content first, the variable content last, and never interleave them. We covered the application-side version of this in reducing AI agent cost with plan caching. If you do one thing after reading this post, do this one.
Lever Three: Speculative Decoding, and Why It Backfires at Scale
Speculative decoding, introduced by Leviathan and colleagues at ICML 2023, uses a small draft model to guess the next k tokens, then verifies all of them in a single forward pass of the large model. Accepted guesses are free. Rejected ones cost a wasted verification. Reported speedups land in the 2x to 3x range for the original formulation, and newer methods like EAGLE, which drafts in feature space rather than token space, push higher.
Here is the part that gets missed in vendor benchmarks.
Speculative decoding does not create throughput. It converts idle compute into lower latency. It works because decode at small batch sizes leaves the tensor cores nearly empty, so verifying five candidate tokens costs almost the same wall clock as verifying one. You are spending FLOPs you were not using.
Once your batch size is large enough that compute is the actual constraint, that spare capacity is gone. Now every speculative token you verify competes with real user requests for the same FLOPs, and the rejected drafts are pure waste. There is a crossover batch size above which speculation reduces total system throughput. Where it sits depends on your model size, draft acceptance rate, and hardware, but it exists, and it is usually lower than teams expect.
The practical rule:
| Situation | Speculative decoding verdict |
|---|---|
| Low traffic, latency-sensitive, single-user sessions | Strong win, turn it on |
| Long decode phases (report generation, long-form writing) | Strong win |
| Agent steps with 80 token outputs | Marginal, the decode phase is too short to amortize draft overhead |
| High-throughput batch processing at saturation | Likely negative, measure before enabling |
| Cost-optimized offline workloads | Usually wrong lever, raise batch size instead |
Acceptance rate is the variable that decides everything, and it is domain dependent. A draft model that guesses well on English prose may guess badly on your JSON tool calls or your codebase. Measure acceptance on your actual traffic, not on a public benchmark. Below roughly 60 percent acceptance, the bookkeeping usually eats the gain.
The counter-intuitive summary: speculative decoding is a latency feature that is marketed as a cost feature, and it helps least on exactly the agent-shaped traffic that is growing fastest.
What to Do Monday Morning
A sequence that works, in order of effort:
- Instrument token shape. Log input tokens, output tokens, and prefix hash per request for one representative week. Compute median P, median D, and R.
- Order your prompts for caching. Stable prefix first, variable content last. Verify cache hit rate went up. This is often a 40 to 80 percent prefill reduction on agent traffic for an afternoon of work.
- Turn on chunked prefill and continuous batching. Confirm your p99 inter-token latency drops. If it does not, your bottleneck is elsewhere.
- Set separate service levels. Time-to-first-token and time-per-output-token are different promises with different fixes. Measuring them as one blended latency number hides which lever to pull.
- Benchmark with your own token shape. Public numbers, including MLPerf Inference, use standardized input and output lengths. They are useful for comparing hardware against hardware, not for predicting your bill. Replay your actual traffic distribution.
- Evaluate speculative decoding at your real batch sizes, with your real prompts, measuring acceptance rate.
- Consider disaggregation last, when you are past a few nodes, your ratio is heavily prefill-weighted, and steps 2 through 4 are exhausted.
If you are on an API provider rather than self-hosting, steps 1, 2, and 4 still apply directly. Input and output tokens are priced differently because the underlying economics are different, and cached input tokens are priced differently again. Your token shape is still the number that determines your bill, which is the same argument we made about open-model pricing in the tokenomics of agent economics.
How OpenNash Can Help
Most of the value in this work is in the measurement, not the exotic infrastructure. When we audit an agent deployment, the first artifact is a token shape profile per workflow, because it usually reveals that two or three agents account for most of the spend and that they share a prompt prefix nobody thought to cache.
From there the path is ordinary engineering: reorder prompts for cache hits, split latency service levels so the team stops optimizing a blended number, benchmark serving configurations against replayed production traffic rather than synthetic prompts, and only then evaluate whether a disaggregated pool earns its operational complexity. Everything ships with the instrumentation attached and stays with your team.
Be clear about when this is not for you. If you are running under a few million tokens a day, use a managed API and spend your time on the agent logic. If you have an existing platform team with serving expertise, they should own this and you should send them the Sarathi-Serve and DistServe papers. Custom work makes sense when agent traffic is material, the token shape is far from chat-shaped, and nobody internally owns the serving layer.
Book a call to map your token shape to a serving configuration.
The teams getting inference costs right in 2026 are not the ones with the best vendor discount. They are the ones who measured their prefill:decode ratio, discovered it was 200:1, and stopped configuring their stack for a workload they no longer run.