Measuring AI Coding Assistant Productivity: Throughput, Revert Rate, and the Metrics That Do Not Lie

In July 2025, METR ran a randomized controlled trial that should have ended the acceptance-rate era. Sixteen experienced open-source developers worked through 246 real issues on repositories they already maintained, randomly assigned to allow or disallow AI tools. The developers predicted AI would make them 20 percent faster. Afterward, they reported it had made them 20 percent faster. The measured result was a 19 percent slowdown.

That gap is the whole problem in one number. Not that AI coding tools are useless, because plenty of other studies find real speedups. The problem is that the people using the tools, including the engineering leaders buying them, cannot tell the difference between feeling fast and being fast. And the metrics most teams have instrumented are built to reinforce the feeling.

Your CFO is going to ask what the per-seat and per-token spend bought. "Eighty-two percent of our engineers use Claude Code daily" is an answer about adoption. It is not an answer about output.

Why Acceptance Rate Is a Vanity Metric

Acceptance rate is the share of AI suggestions a developer keeps. Vendors report it because it goes up and to the right, and because it is the only signal available inside the editor. It has three defects that make it useless for a spend justification.

It measures a keystroke, not an outcome. A suggestion accepted at 10:14am, rewritten at 10:22am, reviewed for forty minutes, merged, and reverted the following Tuesday scores identically to a suggestion that shipped clean. The metric closes its books at the moment of acceptance, which is the moment before all the interesting information arrives.

It is trivially inflatable. Turn down suggestion aggressiveness and acceptance goes up. Restrict the agent to boilerplate and acceptance goes up. Neither change delivers a single additional feature.

It ignores what the acceptance cost. Review is where AI-authored code sends the bill. Reviewers are reading more code, written by someone who cannot be asked a follow-up question, in a style that is often plausible and subtly wrong. Stack Overflow's 2025 survey found 66 percent of developers frustrated by AI solutions that are "almost right, but not quite", with 45 percent saying debugging AI-generated code takes longer than expected. That time is real and it lands on your senior engineers.

Lines generated is worse. It has all of acceptance rate's problems plus an active incentive to produce more code, which is the opposite of what a healthy codebase wants. GitClear's analysis of hundreds of millions of changed lines found duplicated code blocks rising sharply while refactored and moved code declined, a signature of code being added faster than it is being consolidated.

The Four Metrics That Hold Up

Here is the set worth building. Each one can be computed from version control and CI telemetry you already have, and none of them can be gamed by pressing Tab more often.

Revert rate and rework ratio are complements, not alternatives. Most bad code is never reverted. It gets patched by the next person who touches the file, which looks like normal development unless you are attributing changed lines back to their origin. A team measuring only reverts will conclude their agents are fine while the rework ratio quietly runs at double the human baseline.

Review time per pull request is the one engineering leaders skip and then regret. If agent-authored throughput rises 30 percent while reviewer-active minutes per PR rise 50 percent, you have not automated work. You have moved it from the people who enjoy it to the people who are your bottleneck.

Change failure rate ties the whole thing to something the business already understands. DORA's 2024 research found that each 25 percent increase in AI adoption was associated with a 7.2 percent decrease in delivery stability and a 1.5 percent decrease in throughput, which is a striking result given how confidently the same teams reported feeling more productive. DORA's later work reframed this: AI amplifies whatever your delivery system already is. Teams with strong test coverage, small batch sizes, and fast feedback get faster. Teams without those get faster at producing defects.

The mental model: Net Accepted Change

One number to carry into the budget meeting:

Net Accepted Change = merged pull requests that survive 14 days without revert and without rework above threshold, per engineer-week, held constant on task class.

It is deliberately harsh. A PR that ships and then needs three follow-up fixes does not count. A PR that ships clean counts once regardless of whether it was 40 lines or 400. This strips out the two easiest ways to fake progress: shipping volume and shipping fast at someone else's expense.

How to Run the Comparison Honestly

The comparison you want is agent-authored versus human-authored changes, on the same repository, in the same period, on the same class of work. Anything else imports confounders you cannot untangle later.

Tag at the source. Configure the harness to write a commit trailer on every commit it authors:

X-Agent: claude-code/2.1
X-Agent-Model: claude-opus-5
X-Agent-Autonomy: supervised

CI mirrors this into a pull request label. Three autonomy tiers are enough: suggested (inline completion, human drives), supervised (agent drafts the change, human reviews before commit), and delegated (agent runs a full task loop and opens the PR). These behave differently enough that averaging them together destroys the signal. If you have been building around agent harnesses rather than agent prompts, this tagging is a small addition to work you have already done - see our notes on writing harnesses instead of writing code.

Never rely on retrospective self-reporting. Developers do not remember which lines came from where, and the METR result suggests their recall would be biased even if they did.

Control for task class, not for lines. Complexity adjustment is where most internal studies fall apart. Lines changed is not complexity; a 600-line dependency bump is trivial and a 12-line change to a payment retry path is not. Bucket work by something structural:

• Blast radius: number of distinct modules and services touched by the diff
• Historical defect density of the files touched, computed from the last 12 months of fix commits
• Test surface: whether the change is covered by existing integration tests or requires new ones

Then compare within buckets. Agents take a disproportionate share of low-blast-radius work, so an unadjusted comparison will make them look excellent on revert rate and terrible on nothing. That is not a finding, it is a sampling artifact.

Do not measure for less than eight weeks. Revert-within-14-days needs a 14-day tail, and the first two weeks of any tool rollout are contaminated by novelty and by developers reaching for the agent on tasks it is bad at. A 12-week window with the first two discarded is a defensible design.

What the Numbers Actually Look Like

The research does not support a single verdict, which is itself the finding.

The strongest positive results come from constrained, greenfield tasks. The 2023 GitHub Copilot experiment had developers write an HTTP server in JavaScript and found the treatment group finished 55.8 percent faster. Google's enterprise randomized trial, closer to production conditions, found a 21 percent reduction in time to complete a defined development task, which is a large and credible effect.

The negative results come from experienced engineers working in large, familiar codebases with high correctness requirements. That is the METR condition, and it is also the condition most of your senior engineers work in most of the time.

The reconciliation is not "AI good" or "AI bad." It is that the productivity delta is a function of codebase familiarity, task novelty, and correctness cost, and those vary enormously across the same engineering organization. Which means an org-wide average is close to meaningless. Measure by team and by task class or do not bother.

A composite pattern from instrumented teams, offered as a shape rather than a benchmark: delegated agent PRs on well-tested service code often show throughput gains in the 20 to 35 percent range with revert rates within noise of human baseline. The same setup pointed at legacy code with thin test coverage tends to show flat or negative net accepted change, because rework ratio climbs enough to eat the gains. The differentiating variable is almost always test coverage, which is a very old answer to a very new question.

Two failure signatures worth watching for specifically:

Review-time inversion. Median review time on agent PRs exceeds human PRs by more than 40 percent. This means reviewers do not trust the changes and are reading line by line. Either the agent needs tighter task scoping or your review process needs a different shape for machine-authored diffs.

The rework cliff. Revert rate looks fine, rework ratio is 2x human baseline. The agent produces code that passes review and tests, then turns out to be structurally wrong in ways discovered three weeks later. This is the expensive failure mode, and it is invisible to every metric a vendor dashboard exposes.

Building the Instrumentation

None of this needs a platform purchase. It needs a scheduled job and a table.

1. Week 1: tag. Add commit trailers in the harness config. Add a CI step that reads trailers and applies PR labels. Backfill is not possible, so this gates everything else.
2. Week 2: extract. Nightly job pulling merged PRs, their labels, diff stats, review timestamps, and reviewer identities into a warehouse table. One row per PR.
3. Week 3: revert detection. Match Revert "..." commits back to originating PRs. Catch the manual cases by matching diffs that invert a prior diff on the same file set within the window.
4. Week 4: rework attribution. For every merged PR, run blame over subsequent commits touching the same files, attribute changed lines to their originating PR, and compute the 21-day rework ratio.
5. Week 5: task classification. Compute blast radius and historical defect density per PR from repository history. Bucket into three or four classes.
6. Weeks 6 onward: read nothing. Let it run. Interpreting a four-week series will produce a conclusion you have to walk back.

The SPACE framework's core argument holds here: productivity is multidimensional and no single metric captures it. Pair the delivery numbers above with a quarterly developer experience survey asking specifically about trust in agent output and time spent debugging it. When those two data sources disagree, the disagreement is the finding. The METR participants' 39-point gap between perceived and actual speed is the reason you collect both.

This is the same discipline as evaluation-driven development for AI features, applied to the engineering process instead of the product. You are running evals on your own delivery system.

How OpenNash Can Help

Most teams already have the raw telemetry sitting in GitHub and CI and have never joined it into something a finance conversation can survive. OpenNash builds that pipeline as a production system: harness-level tagging, warehouse extraction, revert and rework attribution, and a dashboard split by authorship class and task cohort, with the whole thing handed over as code you own.

The honest guidance on whether you need it: if you have fewer than roughly 15 engineers or under a few hundred merged PRs a quarter, your sample sizes will not support the comparison and you should read individual PRs instead. If your agent usage is entirely inline completion, the measurement problem is genuinely hard and possibly not worth solving yet. If you are running delegated agents that open their own pull requests at meaningful volume, you are already accumulating a quality liability you cannot see, and instrumentation pays for itself the first time it catches a rework cliff.

Book a call to map this measurement stack to your repository and delivery process.

The teams that get this right are not the ones with the best dashboards. They are the ones willing to publish a number that says the tool did not help on a particular class of work, and then narrow the tool's scope accordingly. Every organization measuring acceptance rate is, by construction, unable to discover that. Start tagging commits this week, and in eight weeks you will know something your vendor cannot tell you.