Quantized Model Tool Calling Accuracy: Where FP8 and INT4 Break Agents

The GPTQ paper showed in 2022 that GPT-style models with 175 billion parameters could be compressed to 3 or 4 bits per weight in about four GPU hours with negligible accuracy loss. Since then, "negligible accuracy loss" has become the default assumption for anyone self-hosting a model. Teams pull an AWQ or FP8 checkpoint, see that its MMLU score sits within a point of the original, and put it behind their agent. Then the agent starts calling update_ticket with an order ID that has two digits transposed, and nothing on the leaderboard explains why.

This post makes one argument. A quantized model can match the original at the token level without matching it at the trajectory level. If your system calls tools, you need to re-run your workflow evals for every quantization scheme you deploy, because the damage shows up as wrong tool arguments, not as lower benchmark scores.

Why Cost Pressure Is Pushing Agents to 4-Bit Without Testing

The case for quantization is strong, and nothing here argues against it. Halving the bytes per weight roughly halves the memory needed to hold a model. That frees room for KV cache and larger batches, and on memory-bound decode it often means faster tokens. A Devoxx Greece 2026 talk on quantization and agentic AI goes further and frames quantization as what makes multi-model agent systems affordable on limited hardware. Several small specialized models can beat one large model when the compute budget is fixed.

The serving stack makes this easy. The vLLM quantization docs list support for AWQ, GPTQ, FP8, INT8 W8A8, bitsandbytes, GGUF, and a quantized KV cache, among others. Switching schemes is usually a flag or a different checkpoint name. When a change is that cheap, people stop testing it.

The validation gap is where the trouble starts. Quantized checkpoints on public model hubs often come with recovery tables, and those tables tend to cover knowledge, reasoning, and math benchmarks. Few include a function-calling row. Almost none include a multi-step trajectory metric. Teams read "99% recovery" as "99% as good for my use case," but the number only applies to the tasks in the table.

If you read our earlier comparison of vLLM, SGLang, and TensorRT-LLM under agentic load, this is the precision version of the same problem. Throughput benchmarks don't describe agent traffic, and quality benchmarks don't describe agent behavior.

The Mechanism: What Quantization Does to an Agent

Quantization maps high-precision values onto a coarser grid, which adds noise to every matrix multiplication. The research history explains why that noise is uneven.

• Outlier features. The LLM.int8() paper found that large transformers develop a small number of activation dimensions with extreme magnitudes, and that naive 8-bit quantization breaks when those outliers get crushed. Its fix was to keep them in higher precision.
• Salient weights. AWQ showed that protecting a small fraction of salient weights, about 1%, chosen by activation magnitude, greatly reduces quantization error. Most weights tolerate rounding. A few carry far more of the model's behavior.
• Calibration data. GPTQ and AWQ both use a calibration set to decide how to round. If that set is generic web text, the rounding is tuned for generic web text, and nothing tunes it for JSON tool schemas or 40-step trajectories.

Now apply that noise to the kinds of output an agent produces.

A knowledge benchmark lives in the first two rows. An agent lives in the last five. When the correct token wins by a wide margin, a little noise changes nothing. When the model has to choose between refund_partial and refund_full, or copy ORD-88241907 from a tool result 20,000 tokens earlier, the margin is thin and noise decides more outcomes.

Long context makes this worse. Copying an identifier depends on attention finding the right position, and KV cache quantization, which vLLM exposes as a separate FP8 option, lowers the precision of exactly the keys and values that attention searches over. You can quantize weights carefully and then lose accuracy on recall through the cache setting.

Code generation is the closest well-studied analogue, since a single wrong token can break the whole output. A 2026 ICPC study, Evaluating the Impact of Post-Training Quantization on Large Language Models for Code Generation, looks at this directly. Tool calling has the same property: a function call with one wrong field is a wrong function call.

Compounding: Why Small Deltas Become Large Failures

Agents chain decisions, and per-step error compounds across the chain. Here is an illustrative calculation (the numbers are made up to show the shape, not measured):

A two-point drop in per-step accuracy, which would look like noise on a leaderboard, takes a 20-step workflow from about 82% completion to about 54%. Single-turn evals can't see this. Each step looks almost as good as the baseline, and the cost shows up only when the steps are multiplied together.

Real trajectories don't fail as cleanly as the table suggests. Agents sometimes recover from a bad call when the tool returns an error, and sometimes a wrong argument succeeds silently and corrupts a record. That second case is the one to worry about. We covered why averaged metrics hide it in Agent Run Variance: Why pass@k Hides the Failures That Matter. A quantized model that succeeds on retry still looks fine in pass@k, while it doubles your tool calls and leaves a few wrong writes in your CRM.

Also watch the trajectory length distribution. Lower-precision models sometimes take more steps on the same task, re-reading tool output or retrying calls. If you only track success rate, you'll miss that, and then the cost savings you expected from quantization disappear into extra tokens.

Constrained Decoding Changes the Failure Mode Without Removing It

This is where the problem gets less obvious. Most production agent stacks now use structured output or grammar-constrained decoding, which masks invalid tokens so the model can only emit JSON that matches the tool schema. It's a good practice, and our post on output contracts and schema validation argues it's the cheapest reliability layer you can add.

Constrained decoding does change what quantization damage looks like in your logs. Without constraints, a degraded model emits broken JSON, the parser throws, and you get a loud error. With constraints, the grammar forces a valid object, so the model's uncertainty goes into the field values instead. The enum is valid but wrong. The ID has the right format but points to a different customer. The date parses but has month and day swapped.

As a result, a quantized deployment with constrained decoding can show a 0% schema violation rate and a worse task success rate than the BF16 baseline. If your dashboard tracks parse failures as the main tool-calling health metric, it'll report quantization as free.

Add these checks on top of schema validation:

• Referential checks. Every ID in a tool argument has to appear in the context the agent has seen (a prior tool result, the user message, or retrieved documents). This one deterministic check catches most transposed and invented identifiers.
• Value-range checks. Refund amounts can't exceed order totals, and dates have to fall inside plausible windows.
• Enum confusion logging. Record which enum values the model picks for each task type. A shift in that distribution between BF16 and INT4 is an early signal.

A Per-Scheme Evaluation Protocol

Treat each quantization scheme as a separate model release. Each one needs its own eval run, its own baseline comparison, and its own sign-off. This protocol works for most self-hosted agent stacks.

1. Fix the baseline first. Serve the BF16 (or the original released precision) model on the same inference server, with the same batch size, the same sampling settings, and the same constrained-decoding configuration you'll use in production. Batch composition alone can move scores at temperature 0, as we described in LLM Nondeterminism at Temperature 0. If the baseline runs on a different server setup, you'll be measuring the setup along with the precision.

2. Build the eval set from real trajectories. Take 100 to 300 tasks from production traces or a realistic simulation. Over-sample tasks with long context, many steps, similar enum values, and IDs that have to be copied from deep in the context. Public suites like the Berkeley Function Calling Leaderboard are useful as a smoke test, but they don't contain your tools, your schemas, or your data.

3. Test each scheme on its own, then in combination. Run each of these separately:

4. Run multiple trials per task. At least five runs per task per variant. A single run can't tell a real regression apart from sampling variance.

5. Measure at three levels.

• Call level: schema-valid rate, argument exact-match against a reference, identifier copy accuracy, enum accuracy.
• Trajectory level: task success, steps per task, tool errors per task, recovery rate after a tool error.
• Economic level: tokens per successful task, GPU-seconds per successful task, and escalations to a human.

6. Set the gate before you look at results. For example, a variant ships only if task success is within two points of the baseline and identifier copy accuracy shows no measurable drop. Decide this ahead of time so the cost savings can't talk you into accepting a regression afterward.

Cost Per Successful Task, Not Cost Per Token

Quantization reduces cost per token, but the business pays per completed task. Here is an illustrative comparison (the numbers are hypothetical and chosen to show the arithmetic):

In this example, INT4 halves the compute bill and still costs 70% more per task, because human cleanup costs far more than GPU time. Reverse the inputs (cheap failures, expensive compute, short trajectories) and INT4 wins. The arithmetic is simple. The risk is skipping it and assuming that token savings automatically mean workflow savings.

The table also leaves out silent wrong writes, which don't create a handling cost until someone finds them, possibly weeks later. For agents that write to systems of record, put a higher weight on those failures than the table does.

Quantize by Component, Not by Deployment

An agent system is usually several model roles sharing one serving budget, and each role tolerates precision loss differently. This mapping is a reasonable starting point to confirm with your own evals:

This matches the multi-model pattern the Devoxx talk describes. Small quantized models can handle the high-volume, error-tolerant steps, while precision goes to the steps where one wrong token becomes a wrong action. Many teams find that the router and summarizer account for most of their tokens, so quantizing only those components gets a large share of the savings without risking the tool-calling path.

Keep KV cache precision as its own decision. If the planner works over long contexts with identifiers scattered through tool results, test the FP8 KV cache separately before enabling it, even when FP8 weights passed.

Running This Before the Next Checkpoint Swap

OpenNash builds self-hosted agent stacks where the precision decision goes through the eval harness instead of a config flag. In practice, that means the components described above. We capture production trajectories as a replayable eval set, run each quantization variant against a matched BF16 baseline, add referential and value-range checks behind the schema validator, and gate each scheme on cost per successful task. The client owns the harness after handoff, so the next checkpoint upgrade or precision change runs through the same gate without needing us.

This work matters most for teams running their own inference on a fixed GPU budget, especially when agents write to CRM, ERP, or ticketing systems. If you're calling a hosted frontier API, the provider's serving precision is out of your hands, and your effort is better spent on output contracts and trajectory evals. If your agent is single-step and read-only, quantize aggressively and spot-check.

If you're about to move an agent from BF16 to FP8 or INT4, book a call with OpenNash and bring a week of production traces. We'll scope which components can drop precision safely and what the per-scheme eval gate should be before anything ships.