Choosing an Inference Server: vLLM vs SGLang vs TensorRT-LLM Under Agentic Load

A team I talked with last quarter spent three weeks benchmarking inference servers. They ran the standard sweep: ShareGPT prompts, batch sizes from 1 to 256, output length 256 tokens, three servers, one 8xH100 node. TensorRT-LLM won on tokens per second. They built the engine, shipped it, and put their research agent behind it.

Production p95 for a full agent task came in at 41 seconds against a 15 second target. The GPUs were at 30 percent utilization. Nothing was saturated. The benchmark had been correct and completely irrelevant.

The mismatch is structural. Chat benchmarks measure a workload where each request carries a fresh prompt and produces a long generation. Agent traffic does the opposite: the same enormous prefix arrives over and over, and each round trip produces forty tokens before the model stops to call a tool. If you pick a server on a chart built from the first shape, you are optimizing a variable that barely appears in your bill.

Agent traffic has a different shape, and the difference is measurable

Four properties separate agentic load from chat load. Each one changes which server wins.

Long shared prefixes. A production agent carries a system prompt, tool schemas, few-shot examples, and retrieved context. Eight to twenty thousand tokens of prefix before the user says anything is normal. Every request in the fleet shares most of it. In chat, prefixes are short and mostly unique.

Bursty tool-call round trips. One user task becomes eight to fifteen model calls, separated by tool execution that takes anywhere from 50 milliseconds to 8 seconds. The server sees a burst, then silence, then another burst, with the same conversation coming back with a slightly longer context each time.

Short decodes. The model emits a tool call and stops. Median output length in the agent loops we instrument sits between 30 and 150 tokens. Chat benchmarks assume 256 to 1,024. When decodes are short, per-request overhead and scheduling latency stop being rounding errors.

Latency compounds multiplicatively. A 400 millisecond time-to-first-token penalty is invisible in chat. Multiply it by twelve round trips and you have added 4.8 seconds to a single user-facing task. This is the part that surprises people who moved from a chatbot to an agent on the same infrastructure.

The practical consequence: your throughput chart is measuring decode efficiency on a workload that is 70 percent decode. Your agent workload is 60 to 85 percent prefill by token count. Different bottleneck, different winner.

The prefill math that decides your bill

Take an agent with an 8,000 token static prefix (system prompt plus tool schemas plus a few examples). Each turn appends roughly 600 tokens of tool result and model output. Run twelve turns.

Without prefix reuse, every turn re-prefills the entire accumulated context:

Turn 1:  8,000 tokens
Turn 2:  8,600
Turn 3:  9,200
...
Turn 12: 14,600
Total:   135,600 prefill tokens

With working prefix reuse, you prefill the static block once and then only the delta:

Static prefix:     8,000
12 x 600 delta:    7,200
Total:            15,200 prefill tokens

That is 8.9x less prefill compute for identical user-visible behavior. On a 70B-class model where prefill dominates your GPU-seconds, this is the difference between one node and nine.

The memory side sets the ceiling. For a Llama-3-70B-shaped model (80 layers, 8 KV heads with grouped-query attention, head dimension 128, FP16), each cached token costs:

2 (K and V) x 80 layers x 8 heads x 128 dim x 2 bytes = 320 KB per token

An 8,000 token shared prefix is 2.5 GB of KV cache. If it is genuinely shared across every concurrent request, you pay for it once. If your server evicts it under pressure, you pay 2.5 GB of prefill for every request that misses. On an 8B model the same prefix costs about 1 GB, which is why small-model agent fleets tolerate much sloppier cache management and why lessons do not transfer up.

This is the number to instrument first. Token-weighted prefix cache hit rate predicts your cost curve better than any throughput benchmark you will run.

What each server actually optimizes for

Paged KV memory is table stakes now. The PagedAttention paper that introduced vLLM in 2023 solved fragmentation by borrowing virtual memory paging, and every serious server implements some version of it. The differentiation moved up the stack to cache policy and scheduling.

SGLang is the one built for the traffic pattern agents produce. RadixAttention keeps cached prefixes in a radix tree rather than a flat hash table, so a hundred conversations that share a system prompt but diverge after turn three share the trunk automatically. Its structured generation compiler also matters more than teams expect: if your agent's tool calls are constrained JSON, having the grammar enforced in the decoding loop removes a class of retry that otherwise multiplies your token spend. The SGLang docs are honest about the hardware coverage trade.

vLLM is the default for a reason that has nothing to do with peak numbers. Model support lands there first, the community is large enough that your production bug has probably been filed already, and automatic prefix caching is on by default in recent releases. If your agent stack changes models every six weeks, the operational tax of anything else usually exceeds the performance gap. The vLLM documentation on automatic prefix caching and chunked prefill is the reference to read before you tune anything.

TensorRT-LLM wins the benchmark and loses the iteration loop. Kernel fusion, FP8 and FP4 quantization paths, and in-flight batching tuned against specific NVIDIA silicon produce real throughput advantages. But you compile an engine per model, per parallelism configuration, per precision, and per max sequence length. Change your context window and rebuild. The project repository documents the workflow clearly. For a stable, high-volume, single-model deployment where GPU cost is the binding constraint, that build tax amortizes. For an agent platform serving four models that rotate quarterly, it does not.

Scheduler behavior is the second variable nobody benchmarks

Cache hit rate gets you the prefill savings. Scheduler policy determines whether those savings turn into latency your users feel.

The core conflict: a long prefill occupies the GPU and blocks decode steps for every in-flight request. In a chat workload that is one stall. In an agent workload with fifteen concurrent tasks each doing short decodes, one 20,000 token prefill can spike inter-token latency across the whole batch.

Chunked prefill is the fix. The Sarathi-Serve work from Microsoft Research quantified the throughput-latency trade and showed that splitting prefill into chunks scheduled alongside decode tokens materially improves tail latency. All three servers now implement a version of it. It is on by default in some configurations and not others, which is the kind of detail that decides whether your p99 is acceptable.

The settings that actually move agent latency:

• Chunked prefill chunk size. Smaller chunks smooth decode latency and cost some prefill throughput. Start small for agents.
• KV cache memory fraction. Higher fraction means more prefixes stay resident and more requests hit cache. Push it until you see preemption, then back off.
• Max running sequences. Capping concurrency lower than the memory ceiling often improves p95, because queueing beats thrashing when eviction starts.
• Eviction policy. LRU over a radix tree behaves differently from LRU over flat blocks when your traffic has a bursty arrival pattern with idle gaps between tool calls. This is where agent traffic breaks assumptions built for chat.

That last point is underrated. Between tool calls, an agent conversation is idle for seconds. If the server evicts its prefix during that gap, the next round trip pays full prefill. Some deployments improve p95 by 30 percent purely by raising the KV memory fraction so idle conversations survive their tool-execution windows.

Run the benchmark that matches your traffic

Published throughput studies are useful for ruling things out and useless for choosing. Half of them report a single number with no error bars, on a synthetic distribution, at a concurrency level nobody runs. MLPerf Inference is the most rigorous public option and it still does not model tool-call round trips.

The alternative takes about two days.

Step 1: Capture a real trace. Log 500 to 2,000 production agent tasks. For each model call, record arrival timestamp, full prompt token count, shared prefix length, output token count, and the conversation ID linking calls in the same task. If you are not in production yet, capture from staging with your actual system prompt and tool schemas. The prompts matter more than the volume.

Step 2: Replay with real timing. Do not flatten the trace into uniform concurrency. Replay it with the original inter-arrival gaps, including the tool-execution pauses. Those gaps are the thing that stresses cache retention, and removing them produces a benchmark that flatters every server equally.

Step 3: Measure four things per server.

• Token-weighted prefix cache hit rate
• Time to first token at p50, p95, p99
• Inter-token latency at p95 under load
• End-to-end task completion time for the full multi-turn loop

The fourth is the only one your users experience. Report it with a distribution, not a mean.

Step 4: Sweep concurrency until something breaks. Run at 0.5x, 1x, 2x, and 4x your expected peak. Servers rank differently at different load levels, and the crossover point is often inside your normal operating range. A server that wins at 1x and collapses at 2x is a bad choice for a workload with bursty arrivals.

The engineering team at Character.AI published a useful account of how far KV cache design can be pushed when serving is the whole business. Their conclusion generalizes: the wins came from memory layout and cache policy, not from a faster kernel.

A decision rule you can apply this week

If you want a shortcut before running the trace replay:

1. Measure your shared prefix ratio. Take total prefill tokens across a day and divide the portion that is identical across requests. Above 60 percent, cache architecture dominates every other consideration and SGLang deserves a serious look.
2. Count your models. More than two models in rotation, or a model roadmap that changes quarterly, and TensorRT-LLM's build step will cost you more than it saves.
3. Check your output constraints. If every model call must produce valid JSON against a tool schema, native grammar-constrained decoding is worth real throughput. Retry loops are the most expensive thing in an agent stack.
4. Be honest about your team. Two engineers who can debug a scheduler beat a 15 percent throughput edge nobody can maintain. The vLLM community size is a legitimate technical argument.
5. Then replay your trace. The first four steps narrow the field to two candidates. Only the replay picks between them.

How OpenNash Can Help

Most teams do not have a serving problem, they have a measurement problem. The work we do here usually starts with instrumenting the agent loop so prefix cache hit rate, per-turn time to first token, and full-task completion time are visible before anyone touches a server config. That audit typically exposes one or two prompt-structure changes (stabilizing the tool schema block, moving volatile context after the static prefix) that improve cache hit rate more than switching servers would.

From there the pattern is standard: design the serving tier with explicit latency budgets per round trip, build the trace-replay harness so the choice is defensible, deploy with the metrics wired into your existing observability, and hand over the whole thing with documentation. You own the harness and can rerun it when your model changes.

Some teams should not do any of this. If you are under roughly 50 million tokens a month, a hosted API is cheaper than the GPUs plus the engineer-hours, and self-hosting is a distraction. If you are somewhere between and your traffic is growing fast, the honest answer is to instrument now and decide in a quarter with data. Book a call if you want help mapping this to your workflow.

The teams that get this right stop treating inference server selection as a procurement decision with a right answer. It is a fit question between one specific traffic shape and one specific set of scheduling and caching trade-offs. Capture your trace. Replay it. The chart you build in two days will be worth more than every benchmark you read this year.