LLM Nondeterminism at Temperature 0: Batch Effects That Move Your Eval Scores

Thinking Machines Lab sent the same prompt, "Tell me about Richard Feynman", to Qwen3-235B 1,000 times at temperature 0 and got 80 unique completions. All 1,000 matched for the first 102 tokens. At token 103 some completions said Feynman was born in "Queens, New York" and others said "New York City". Nobody changed the prompt, the weights, or the sampling settings. The only thing that varied between requests was what else the server was handling at the time.

That matters for anyone who runs evals. When a prompt tweak moves your benchmark from 81.2% to 82.0% and the team ships on it, you are assuming a rerun of the old system would score 81.2% again. On most serving stacks that assumption has never been tested. The run-to-run noise floor comes from how the server is built, and the sampling settings don't remove it.

Why Greedy Decoding Still Drifts

Temperature 0 means argmax: at every step the model emits the single highest-probability token. That part is deterministic. The logits going into the argmax are not.

Floating-point addition is not associative. In 32-bit floats, (0.1 + 1e20) - 1e20 evaluates to 0, while 0.1 + (1e20 - 1e20) evaluates to 0.1. A transformer forward pass is billions of additions, mostly inside matrix multiplications, normalization layers, and attention reductions. If you change the order of those sums, the last few bits of the logits change. Most of the time that doesn't matter. When the top two candidate tokens are within a rounding error of each other, it decides which one wins, and every later token depends on that choice.

The common explanation blames "GPU concurrency plus floating point", meaning atomic adds that finish in random order. Thinking Machines' write-up pushes back on that. For a fixed batch, a typical LLM forward pass is run-to-run deterministic: feed in exactly the same batch twice and you get the same bits. The drift appears because the batch is never the same twice. Kernels choose different tiling and reduction strategies depending on how many tokens they are processing, so the arithmetic for your sequence depends on how many other sequences are in the batch. Server load sets the batch size, and the batch size sets your logits.

Vincent Schmalbach's breakdown and Michael Brenndoerfer's longer explainer both list more sources that stack on top of this in production APIs:

The table makes one thing clear. The only row that temperature and seed settings fix is the last one, and at temperature 0 that row was already mostly gone.

The Noise Floor Is a Serving Property

If you take one idea from this post, take this one: reproducibility is a property of your serving stack, and you have to measure it per stack. A model behind vLLM on your own H100s, the same weights behind a hosted endpoint, and a frontier API all have different noise floors. The floor also changes with load, so a quiet Sunday benchmark run and a busy Tuesday run of the same configuration are not directly comparable.

Provider seed controls don't change this. OpenAI's cookbook guide to the seed parameter presents it as producing "mostly" consistent outputs and says to watch system_fingerprint, which changes when the backend configuration changes. That fingerprint is useful. Log it with every eval run, and treat a fingerprint change partway through a comparison as grounds to rerun both arms. It does not promise bitwise-identical results within one fingerprint, because batch composition still varies underneath it.

Mixture-of-experts models add another layer. When experts have capacity limits, whether your token goes to its first-choice expert can depend on how many other tokens in the batch wanted the same expert. The Towards Data Science explainer walks through this, and it means MoE models can drift by more than a few low-order bits. A token can take a different path through the network altogether.

This is the same statistical problem we covered in Leaderboard Gaps Smaller Than Error Bars, with a different cause. That post was about item sampling error: a 500-question benchmark has a standard error of about 1.8 points at 80% accuracy (sqrt(0.8 × 0.2 / 500) ≈ 0.018). Batch nondeterminism adds a second error term on top of that, and most teams never measure it.

Measuring Your Reproducibility Floor

The protocol is simple and cheap compared with what a wrong ship decision costs.

1. Freeze everything. Same model version, same prompts, same system prompt, same tool definitions, same decoding settings (temperature 0, fixed seed if supported).
2. Run the full eval N times against the unchanged system. Five runs is a minimum. Ten gives you usable per-item statistics.
3. Vary load on purpose if you control the server. Run some passes alone and some under concurrent traffic, because the floor under load is the one production will see.
4. Record three numbers. These are the spread of the aggregate score, the per-item flip rate (the share of items that were not graded the same way on every run), and the output-level divergence rate (the share of items with more than one distinct output string, even when the grade stayed the same).

Here is a minimal harness for the last step:

import statistics
from collections import defaultdict

def reproducibility_floor(run_results):
    """
    run_results: list of runs; each run is a dict
      {item_id: {"output": str, "passed": bool}}
    """
    n_runs = len(run_results)
    scores = [
        sum(r["passed"] for r in run.values()) / len(run)
        for run in run_results
    ]

    grades = defaultdict(set)
    outputs = defaultdict(set)
    for run in run_results:
        for item_id, r in run.items():
            grades[item_id].add(r["passed"])
            outputs[item_id].add(r["output"])

    n_items = len(grades)
    return {
        "runs": n_runs,
        "score_mean": statistics.mean(scores),
        "score_range": max(scores) - min(scores),
        "score_stdev": statistics.stdev(scores) if n_runs > 1 else 0.0,
        "grade_flip_rate": sum(len(g) > 1 for g in grades.values()) / n_items,
        "output_divergence_rate": sum(len(o) > 1 for o in outputs.values()) / n_items,
    }

The grade flip rate is the most useful of the three. Items that flip are, by definition, items where the model sits on a decision boundary. If a candidate change "fixes" 12 items and 10 of them are in the flip set, you have probably fixed 2. Splitting results into stable items and flipping items, and reporting deltas only on the stable set (with the flipping set reported separately), turns a noisy aggregate into something you can reason about.

Agents make this worse. A flipped token in step 2 of a 15-step trajectory can send the agent down a different tool path entirely, so agent evals often show much higher flip rates than single-turn evals on the same model. Agent Run Variance covers why pass@k hides this, and the same repeated-run discipline applies here.

Gating Deploys on Deltas You Can Trust

Once you have a floor, write it into the deploy gate. An illustrative rule set for a team running a 400-item regression eval might look like this:

These numbers are illustrative. The structure is what matters: a delta with no floor next to it should not count as evidence. In practice, many teams find that a large share of the prompt and harness "improvements" they shipped over a quarter fall inside the inconclusive band. Running five baseline passes and five candidate passes costs ten eval runs instead of two. That is usually cheaper than one incident traced back to a change that never helped. The harness A/B testing post covers how to pair runs so the comparison holds up.

Two practices make the gate hold up in operation:

• Interleave arms in time. Run baseline and candidate passes alternately rather than all-baseline-then-all-candidate, so a load shift or silent provider update hits both arms equally.
• Log the serving context with the score. Store the model ID, system_fingerprint where available, server version, kernel mode, concurrent load, and timestamp. When a score jumps, the first question should be whether the stack changed. Answering it shouldn't take an afternoon.

Batch-Invariant Kernels and What They Cost

If you serve your own models, you can remove most of this noise at the source. The Thinking Machines approach makes the three reduction-heavy operations batch-invariant: RMSNorm, matrix multiplication, and attention. Each sequence's arithmetic then follows a fixed reduction order no matter how many neighbors it has in the batch. With those kernels in place, their 1,000 Feynman completions collapsed to a single unique output.

Throughput is the cost. Batch-dependent strategies exist because they are faster, splitting reductions across more cores when the batch is small, for example. Forcing a single strategy gives some of that up. Their reported numbers show a naive deterministic setup running several times slower than default vLLM, with an improved attention kernel closing much of that gap but not all of it. The Spheron write-up on vLLM batch invariance covers the vLLM side in practical terms, and the vLLM documentation is the place to check which batch-invariant options your installed version supports, since this area has been moving quickly.

Our recommendation is to split the fleet by purpose. Run eval harnesses, regression suites, and trace-replay debugging on batch-invariant kernels, and accept the slowdown, because those workloads need reproducibility more than throughput. Production traffic serving users can stay on the fast kernels, where small numeric drift costs little and throughput has a direct dollar value. If you are choosing a server partly on this basis, Choosing an Inference Server compares vLLM, SGLang, and TensorRT-LLM under agentic load, and determinism support is now a reasonable column to add to that comparison.

This split has a catch. An eval run on deterministic kernels measures a slightly different numeric system from the one serving production. In practice the gap is small, because the kernels compute the same math in a fixed order, but it means production monitoring still needs repeated-run statistics. Deterministic evals tell you whether a change helped. They don't tell you how much production behavior will wobble.

On hosted APIs you have no kernel options. The FutureAGI guide to non-deterministic prompts suggests designing for the variance: repeated sampling, structured outputs that narrow the space of possible divergence, and validators that check what matters downstream instead of exact string matches. Exact-match assertions on free-text LLM output will flake. Schema checks and semantic graders flake far less, because two outputs that differ in phrasing usually agree on the fields that matter.

Reproducibility as an Eval Requirement

For business leaders this looks like an engineering detail, but it sits underneath a governance question. When a team reports that a model upgrade improved accuracy by a point, or that a prompt change cut escalations, the decision to spend money or take on risk rests on that number. If nobody has measured the run-to-run floor, nobody knows whether the number means anything.

The fix is procedural and cheap:

• Every eval report includes the baseline floor (score range across repeated runs) next to the delta.
• Every eval run logs serving context, including provider fingerprint where available.
• Self-hosted eval infrastructure runs batch-invariant kernels by default.
• Any change whose delta falls inside the floor is labeled inconclusive, not shipped as a win.
• Flipping items get reviewed by a human, because they show where the model is uncertain and where production behavior will be least stable.

This fits into the broader eval practice in Production Evals for AI Agents. Repeated-run measurement is the first thing to fix there, because every other metric in the dashboard inherits its noise.

Putting a Floor Under Your Agent Evals

OpenNash builds eval harnesses as part of every agent deployment. Reproducibility measurement is one of the first things we set up, because nothing measured later can be trusted without it. For teams running their own inference, that means configuring a deterministic eval tier next to the production tier and wiring floor measurements into CI. For teams on hosted APIs, it means repeated-run gating, fingerprint logging, and graders that tolerate harmless variation in wording. The harness, the baseline floors, and the gate logic belong to the client after handoff.

If your team has shipped changes on eval deltas under two points and never rerun the baseline, start by rerunning your current production configuration ten times against your existing eval set this week. If the spread is larger than the last improvement you shipped, book a call and we can work through a deterministic eval tier and deploy gate for your stack.