A support agent we audited last quarter scored 94% on the client's internal benchmark. They ran each test case once, saw the number, and shipped. Six weeks later the same agent was closing about four out of five tickets on a quiet morning and closer to two out of three on a Tuesday afternoon when the CRM was slow. The prompt had not changed. The model had not changed. The tools had not changed. The benchmark had simply never measured the thing that broke.
That gap has a name now. A growing body of 2026 agent reliability work argues that rising benchmark accuracy is masking run-to-run instability, and that the metric most teams quote was never designed to catch it. If you are buying, building, or defending an agent this quarter, the single most useful change you can make is to stop reporting a score and start reporting a distribution.
pass@k Was Built for a Problem You Do Not Have
pass@k comes from the Codex paper, where it was defined as the probability that at least one of k sampled programs passes the reference unit tests. In that setting it is an honest metric. You generate twenty candidate functions, the test suite verifies them in milliseconds, you keep the one that compiles and passes, and you throw the rest away. The k attempts are real attempts, and the filter that selects among them is real too.
Now port that metric to a refund agent. Nobody generates five candidate refund decisions and picks the best one. There is no unit test for "did this respond to the customer's actual question." The first trajectory the model samples is the one the customer reads. Under those conditions pass@k describes a workflow that does not exist, and it does so in the flattering direction. A useful walkthrough of how pass@k and mean success rate tell different stories makes the same point from the statistics side: the metric encodes an assumption about the sampling procedure, and if you change the procedure the number stops meaning what you think.
The alternative is pass^k, the probability that all k independent runs succeed. Phil Schmid's breakdown of pass@k versus pass^k for agents frames it as best-case versus consistent-case, which is the right instinct. Operations teams do not care whether the agent can do the task. They care whether it will do the task on Thursday, at 4pm, on the 400th ticket of the day.
The Arithmetic That Makes Optimism Cheap
Take a single per-run success rate, call it p, and look at what the two metrics say about the same agent.
| Per-run success rate | pass@5 | pass^5 |
|---|---|---|
| 70% | 99.8% | 16.8% |
| 80% | 100.0% | 32.8% |
| 90% | 100.0% | 59.0% |
| 95% | 100.0% | 77.4% |
| 99% | 100.0% | 95.1% |
At 90% per run, pass@5 rounds to 100% and pass^5 is 59%. Those two numbers would lead a reasonable executive to opposite decisions, and they describe the same set of runs.
The second piece of arithmetic is worse, and it is the one that catches multi-step agents. Trajectory success compounds across steps:
| Per-step reliability | 5-step workflow | 20-step workflow | 50-step workflow |
|---|---|---|---|
| 95% | 77% | 36% | 8% |
| 98% | 90% | 67% | 36% |
| 99% | 95% | 82% | 61% |
| 99.9% | 99.5% | 98% | 95% |
A 20-step agent built from tool calls that each work 99% of the time completes cleanly 82% of the time. That is not a model problem. It is multiplication. And it explains why so many agent demos that look flawless on a four-step task fall apart when someone asks for the same capability across a real workflow with error handling, lookups, and a write.
The counter-intuitive part: as models improve, this gets harder to see, not easier. METR's work on measuring AI ability to complete long tasks reports time horizons at two thresholds, 50% and 80% success, and the 80% horizon is several times shorter than the 50% one. The headline number moves fast. The reliable-enough-to-ship number moves slower. A vendor quoting the first while you are buying the second is not necessarily lying, but you are having two different conversations.
Where Agent Run Variance Actually Comes From
Most teams assume variance is a prompting problem and try to fix it with better instructions. Some of it is. A meaningful share is not.
Sampling. The obvious one. Temperature above zero means the model draws from a distribution. Lowering it narrows the spread and also narrows the model's ability to recover from a bad intermediate state, so the fix has a cost.
Inference nondeterminism. This is the source that surprises people. Thinking Machines Lab's writeup on defeating nondeterminism in LLM inference traces run-to-run differences at temperature 0 to batch-size-dependent reduction order in GPU kernels. Your request gets batched with whatever else hit the server at that moment, floating-point addition is not associative, and the numerics shift. Identical input, identical settings, different output. You cannot prompt your way out of that, and it means any reliability claim measured on a quiet endpoint may not hold under production load.
Environment flakiness. Tool timeouts, rate limits, a CRM that returns a 502 twice an hour, a stale search index. Discussions around agent benchmarks for enterprise use keep landing on the same finding: harness and environment differences account for a large share of score movement, which is also why SWE-bench results shift between reruns of nominally identical setups.
Context accumulation. Two runs that diverge at step 3 have different context at step 8. Variance compounds inside a trajectory rather than staying constant across it, which is why long-horizon failures cluster instead of scattering.
Retrieval tie-breaking. Near-identical similarity scores resolve in whatever order the index returns them. Reindex and the ordering changes, and so does the answer.
Stochastic agent evaluation is not an exotic research practice. It is the acknowledgment that at least three of these five sources sit outside your prompt file.
Report a Distribution, Not a Score
Here is the reporting format we now require before signing off on any agent that touches a customer or a system of record.
| What to report | Question it answers | How to compute |
|---|---|---|
| Mean pass@1 | Typical single-run outcome | n runs per case, n >= 20, fixed config |
| pass^k | Does it hold for a full shift | k = runs before a human inspects output |
| p5 (worst-case run) | What a bad day looks like | 5th percentile of per-case success across runs |
| Across-seed spread | Is the config or the model unstable | Best-seed minus worst-seed success rate |
| Step-level failure attribution | Where it breaks | Failure count per tool call and per decision node |
| p95 cost and latency | What the tail costs | Per-run token spend and wall clock |
| Retry-adjusted success | Honest first-attempt rate | Count a retried success as a first-attempt failure |
Two rules make the format work. First, twenty runs per case is the floor. Under ten runs you cannot separate a 70% agent from a 90% one, and a single-run demo number carries roughly the information content of a coin flip. Second, report the spread alongside the mean every time. A workflow at 85% mean with a 3-point spread is shippable. A workflow at 90% mean with a 25-point spread between seeds is a system that has not been understood yet, and the higher headline number is actively misleading.
The retry line deserves its own note. Retries are a legitimate mitigation, and internal trace data consistently shows that a second attempt rescues a large share of first-attempt failures. That is exactly why you must record it separately. If your reported success rate quietly includes retry-induced successes, you have hidden the instability inside your latency and cost budget instead of fixing it, and you will discover the tradeoff during your first traffic spike. Track the first-attempt rate as the reliability number, then decide on retries as an explicit engineering choice with a known p95 cost.
Set SLOs on pass^k, Not on Benchmark Accuracy
Once reliability is a distribution, the operating discipline that fits it already exists. Google's SRE practice around service level objectives and error budgets has spent two decades on exactly this problem: pick a target below 100%, spend the remainder deliberately, and stop shipping when the budget is gone.
Work the math from volume backward. A workflow running 500 times a day at a 99% workflow-level target gives you five failures per day of budget. If that workflow has 20 steps, 99% end-to-end requires roughly 99.95% per step, which almost certainly means removing steps rather than improving them. That calculation is the most useful thing an eval suite produces, because it converts "the agent needs to be better" into "the agent needs four fewer tool calls."
Then tier the targets, because uniform reliability requirements are how budgets get wasted on drafting tasks.
| Tier | Example | Reversible | pass^k target | Gate |
|---|---|---|---|---|
| Draft | Reply suggestion, summary | Yes | 95% | Human review by default |
| Classify | Routing, tagging, triage | Cheaply | 98% | Sampled audit |
| Retrieve and answer | Policy lookup, status check | Yes | 98% | Confidence-based escalation |
| Act with approval | Refund, order change | Partly | 99.5% | Mandatory approval |
| Act autonomously | Record write, external send | No | 99.9% | Circuit breaker plus audit log |
The tiering is also the honest answer to "should we automate this yet." A workflow that cannot hit its tier's pass^k after a fair measurement is not ready, and no amount of demo polish changes that. We walk through the operational side of this in more depth in agent reliability engineering and SRE patterns.
Running This Without a Research Team
The measurement loop is smaller than people expect. Six steps, repeatable in a week.
- Pull 30 real cases from logs. Not synthetic ones. Synthetic cases have clean inputs, and clean inputs are where agents look best.
- Run each case 20 times under a frozen config. Same model version, same temperature, same tool endpoints. Record the full trace, not just the final output.
- Compute the distribution table above. Mean, p5, spread, step-level attribution.
- Do error analysis on the failures. Read the traces. Hamel Husain's argument that error analysis is the highest-leverage activity in LLM evals holds even harder for agents, because the failure usually happened four steps before the visible symptom.
- Fix the top failure mode, rerun, compare distributions. If the mean improved and the spread widened, you did not fix anything. You moved the failure.
- Freeze the run as a regression test. Variance regressions are silent otherwise. A model provider ships a point release, your spread doubles, and nothing in your dashboard says so until customers do.
The one habit worth building deliberately: compare distributions, never means. Most agent "improvements" we have measured moved the mean by two or three points and left the tail untouched, which is another way of saying they changed nothing about the runs that generate support tickets. Our approach to building these suites per workflow rather than per model is covered in workflow-specific evals and production evals for agentic systems.
How OpenNash Can Help
We build production agents, so this is not an academic position for us. Every deployment we ship gets a variance profile before it gets a rollout date, and the reliability tier determines the guardrails rather than the other way around.
In practice that means the audit phase pulls real cases from your existing logs and establishes a baseline distribution, the design phase sets pass^k targets and approval gates per workflow tier, and the build phase treats a widening spread as a blocking defect. You own the eval suite, the traces, and the SLO definitions at handoff, because reliability measurement that lives inside a vendor's dashboard is not measurement you can defend to your own auditors.
Platform tools are the right call when your workflow is standard, your volume is moderate, and a 95% tier is genuinely acceptable. Custom is the right call when the workflow is specific to your operation, touches systems of record, or has an irreversible action in it. And if you cannot yet get 30 real cases out of your logs, the right call is to wait a quarter and instrument first. Book a call to map this to your workflow.
Four Questions for Your Next Vendor Demo
Ask these before the deck opens.
- How many times did you run each case, and what is the spread between your best and worst seed?
- Is that number pass@1, pass@k, or pass^k, and what value of k?
- Does it include retries, and what is the first-attempt rate?
- What was the p95 latency and cost on the runs that succeeded, and on the ones that did not?
A team that has done the work answers all four in under a minute. A team that quotes a single number and changes the subject has told you what you needed to know.