Leaderboard Gaps Smaller Than Error Bars: How to Compare Reasoning Models Honestly

A procurement deck crossed my desk earlier this year recommending a model switch. The justification was a single line: the new model scored 2.3 points higher on GPQA Diamond. The contract value was in the low six figures.

GPQA Diamond has 198 questions. At an 80% score, the standard error from sampling alone is about 2.8 points, which puts the 95% interval at roughly plus or minus 5.6 points. The 2.3 point gap that justified the switch was less than half the width of the uncertainty on either model's score. The deck was not wrong about the number. It was wrong about what the number could support.

This happens constantly right now because reasoning models have converged. When the top six models on a leaderboard sit within four points of each other, the rank ordering stops carrying information and starts carrying noise. And almost no public leaderboard publishes the interval that would tell you that.

The arithmetic nobody runs

Benchmark scoring is a binomial process. Each item passes or fails, and the standard error on the resulting proportion follows a formula any spreadsheet can compute: the square root of p(1-p)/n. Multiply by 1.96 for a 95% interval.

Here is what that produces for benchmarks people actually cite in vendor comparisons:

A model reporting 93.3% on AIME and one reporting 90.0% differ by exactly one problem. One. That single problem gets written into press releases, comparison tables, and buying decisions.

Evan Miller's Adding Error Bars to Evals makes the case that this is a solved problem in every other quantitative field and that eval reporting should simply adopt the standard tools: central limit theorem intervals, clustered standard errors when questions come in related groups, and paired differences when comparing models. The paper has been out since late 2024. Most leaderboards still publish bare point estimates.

There is a second variance source stacked on top. Reasoning models sampled at nonzero temperature produce different traces on the same input, so a single run understates the spread. We covered the failure modes this creates in agent run variance and why pass@k hides the failures that matter. For benchmark comparison the practical implication is narrower: a score from one run per item carries run-to-run variance on top of sampling variance, and the reported interval usually accounts for neither.

Four kinds of uncertainty a leaderboard row hides

Sampling error is the easiest to compute and the least interesting. Three others matter more and none of them shrink by adding items.

Harness variance. The same model on the same benchmark scores differently depending on prompt template, few-shot count, answer extraction regex, and whether partial credit is allowed. EleutherAI's Lessons from the Trenches on Reproducible Evaluation of Language Models documents multi-point swings from implementation choices that no paper bothers to report. Mizrahi and colleagues went further in State of What Art? A Call for Multi-Prompt LLM Evaluation, showing that model rankings reorder when you vary the prompt paraphrase while holding everything else fixed. If a ranking flips based on which of five equivalent phrasings you picked, the ranking was never measuring the thing you cared about.

Label error. Benchmarks contain wrong answers. Are We Done with MMLU? audited the dataset and found a meaningful fraction of items with errors, concentrated badly enough in some subsets that a majority of sampled questions in one category were flawed. Epoch AI ran into the same issue with FrontierMath and had to issue corrections, which the FrontierMath v2 analysis walks through. Label error does not average out. It creates a ceiling and rewards models that have memorized the wrong answer alongside everyone else.

Contamination. The cleanest demonstration remains Scale AI's GSM1k work, described in A Careful Examination of Large Language Model Performance on Grade School Arithmetic. They built a fresh test set to the exact specification of GSM8K and re-scored the field. Some model families dropped substantially, and the size of the drop correlated with how likely the benchmark was to appear in that model's training data. Frontier models held up better, but the method is the point: a freshly authored held-out set is the only reliable contamination detector.

Selection and reporting effects. The Leaderboard Illusion examined arena-style evaluation and found that private pre-release testing, selective disclosure of results, and unequal sampling rates give some providers a structural advantage in the final rankings. The scores are real. The comparison is not apples to apples.

Stack these and the honest uncertainty on a public reasoning benchmark score is wider than the sampling interval, often by a lot. Nobody publishes that composite number because nobody knows how to compute it cleanly. That is a reason for humility, not a reason to ignore it.

Paired comparison is the cheap fix

If you are comparing two models rather than reporting one absolute score, the statistics get much friendlier and almost nobody takes advantage of it.

Comparing two independent scores means combining two independent errors. Comparing two models on the same items means you only need to look at where they disagree. Items both models get right, or both get wrong, carry no information about which is better and should not inflate your variance.

McNemar's test formalizes this. Build a 2x2 table of agreement, count the discordant cells, and test whether the split between them departs from chance. On a 200 item benchmark where two models agree on 170 items and split the remaining 30 by 20 to 10, you are running a test on 30 observations, not 200. The interval is still wide, but it is honest and it is a fraction of the width you would get treating the scores as independent.

The practical protocol for a model bake-off:

1. Fix one harness. Same prompt template, same extraction logic, same scoring function, same tool definitions. Version it in git.
2. Run every candidate on identical items. No skipped items, no per-model prompt tuning unless you tune all of them equally and say so.
3. Run each item at least 5 times per model when sampling is nonzero. Report per-item pass rate, not a single binary.
4. Compute the paired difference and its interval, not two separate scores.
5. Cluster your standard errors when items are grouped. Ten questions derived from the same source document are not ten independent observations.

Step five catches people. If your internal eval set has 200 items but they came from 40 support tickets with 5 variations each, your effective sample is closer to 40. Treating it as 200 produces intervals that are too narrow by roughly the square root of the cluster size.

What honest reporting looks like

A few groups do this well and they are worth using as a template for what to demand from vendors.

HELM publishes the full evaluation configuration alongside results, including prompts and per-scenario breakdowns, so you can see what was actually measured rather than a single aggregate. Artificial Analysis publishes methodology and separates quality, speed, and price rather than collapsing everything into one rank. METR's work on measuring AI ability to complete long tasks reports confidence intervals prominently and they are wide, which is the correct and unflattering thing to do. The platinum benchmarks work took the opposite tack: rather than adding items, the authors cleaned label errors out of existing benchmarks and re-tested, finding that frontier models still fail on questions that should be trivial once the noise is removed.

The common thread is disclosure. You do not need a vendor to have narrow intervals. You need them to tell you what the intervals are.

Four questions to put in your evaluation RFP, verbatim:

• How many items were scored, and how many independent runs per item?
• What was the exact harness configuration, including prompt template, few-shot count, and answer extraction rule?
• What is the uncertainty estimate on this score, and does it account for clustering?
• What contamination checks were run, and against what held-out set?

A vendor who cannot answer these is not necessarily hiding anything. Most of them genuinely have not computed it. But the inability to answer tells you the score should not carry weight in your decision, which is itself useful.

Screen publicly, decide privately

The framework we use with clients has two stages and a hard rule between them.

Stage one: screen. Use public leaderboards to go from 20 candidate models to 4. This is exactly what aggregate benchmarks are good for. A model 25 points behind on GPQA is genuinely behind. Coarse filtering on noisy signal works fine because the gaps you are filtering on are much larger than the noise. Cost and latency belong in this stage too, since they eliminate candidates faster than quality does.

The hard rule: no model gets selected at stage one. Public benchmarks screen out. They never screen in.

Stage two: re-rank on your data. Build 150 to 300 items sampled from real traffic, labeled by someone who owns the outcome. Not synthetic. Not sampled from the easy cases. Stratify to include the failure modes you actually see, because those are where models separate. Run all four finalists through one harness, paired, multiple runs per item, and compute the paired difference with an interval.

Two hundred items sounds small. It is small in absolute terms and will not resolve a 1 point difference. But it will reliably resolve a 10 point difference on your distribution, and a 10 point difference on your distribution is worth far more than a 2 point difference on a public set built by someone with different priorities. The model routing decisions that follow from this get much easier when you have a real spread to route against rather than a coin flip.

The counter-intuitive result we see repeatedly: on customer-specific task sets, the public leaderboard rank order holds maybe 60% of the time among the top four. The remaining 40% is where the money is. A model ranked third publicly that wins by 12 points on your ticket triage distribution is worth switching to. A model ranked first publicly that wins by 1.5 points is not worth the migration cost, because 1.5 points is not a real number at that sample size.

When the gaps really are inside the error bars

Sometimes the honest answer after a proper bake-off is that two models are indistinguishable on your tasks. That is a finding, not a failure. It means the decision should be made on the things you can measure precisely: cost per resolved task, p95 latency, rate limits, data residency, contract terms, and how quickly the provider deprecates versions. Those numbers have tight intervals and real business consequences. The API pricing dynamics across providers move faster than capability gaps do right now, and a 3x price difference is a much stronger signal than a 2 point benchmark difference.

It also means you should build for substitution. If two models are statistically tied today, one of them will be ahead in six months and it will not be the one you guessed. Abstract the model behind an interface, keep the eval set running on a schedule, and make switching a config change rather than a project.

How OpenNash Can Help

Most teams we work with have a model selection problem downstream of an evaluation problem. They do not have a labeled task set, so they fall back on leaderboards, and the leaderboards cannot answer the question they are asking.

The work is unglamorous and it is the whole game. We audit the actual task distribution from production logs, stratify a 150 to 300 item eval set that includes the failures rather than just the happy path, build a fixed harness with versioned prompts and deterministic scoring, and run paired comparisons with real intervals. The output is a ranking you can defend in a procurement review, plus a regression suite that keeps running after the decision is made. Clients own the harness and the data outright.

That eval work also feeds directly into deployment: the same fixed harness becomes the gate on prompt changes, model upgrades, and tool additions. Our broader approach to this is in how to eval AI agents in 2026.

If you are being asked to justify a model decision on a gap you suspect is inside the noise, book a call and we will map this to your workflow.

The one habit worth building

Before any benchmark number enters a decision, compute the interval. It takes thirty seconds: square root of p(1-p)/n, times 1.96. Write it next to the score.

Do this for a month and the effect is immediate. Most of the comparisons you were about to make will visibly dissolve, and the few that survive will be the ones worth acting on.