Reviewing AI Generated Code at Scale: Redesigning the Merge Path

A staff engineer I worked with last quarter opened her review queue on a Monday and found 41 pull requests. Six were hers from before the weekend. The other 35 came from three coding agents that had run overnight against a backlog of tickets. Every diff was plausible. Every one had tests. She approved 22 of them by lunch.

Ask her what was in diff number 17 and she cannot tell you. Neither can anyone else on that team. The code is in production.

This is the part of the agentic coding story that does not show up in the demo videos. Generation got cheap and verification did not, and the gap between them is now where delivery actually dies.

The Queue Math Nobody Ran

Merge throughput is not the sum of what your engineers and agents can write. It is the minimum of what can be written and what can be approved. Once agents cross from suggestion to authorship, generation stops being the constraint and review becomes the only stage that matters.

The unpleasant part is how the wait time behaves. Treat the review queue as a single-server system with utilization ρ (arrival rate divided by review capacity). Average queue wait scales as ρ / (1 - ρ) multiplied by the time a single review takes:

A team where reviewers were comfortably at 50% utilization before agents, and where agents doubled the inbound diff count, lands at 100% utilization. That is not "twice as slow." That is a queue with no steady state, which is exactly what a 41-item Monday morning looks like.

Two escape hatches exist. Add review capacity, which does not work because senior judgment is the scarce input and you cannot hire it at agent speed. Or reduce the review work per change. Everything useful lives in the second option.

The 2024 and 2025 DORA research points the same direction from a different angle: teams adopting AI tooling report throughput gains alongside flat or degraded delivery stability. The generated code is not necessarily worse. The verification step around it did not scale, so defects escape at a higher rate. AI amplifies whatever your delivery system already was, including the parts that were held together by one person reading carefully.

Line Reading Was Always a Weak Control

Before redesigning review, it is worth being honest about how well the old process worked, because the nostalgia is doing a lot of unearned work.

Microsoft Research's study of expectations and outcomes in modern code review found that while developers rank defect finding as the top goal, the actual output of most reviews is knowledge transfer and small code improvements. Google's own case study of code review at scale found the median change is small, has one reviewer, and clears quickly. That process works because changes are tiny and reviewers know the surrounding code.

Both preconditions break with agent authorship. Diffs get large because the marginal cost of writing 800 lines is the same as 80. And nobody on the team has the context of having written it, because nobody wrote it.

The classic SmartBear and Cisco review data pegged effective review at roughly 200 to 400 lines in a 60 minute sitting, with defect detection falling off sharply past that. Meta hit the same wall from the latency side and published on reducing code review time by attacking diff size and reviewer routing rather than asking people to read faster.

So the honest framing: line-by-line review was a heuristic that worked when a human author had already done the reasoning and the reviewer was sampling for mistakes. Agent-authored code removes the author's reasoning from the room. Sampling a diff you have no model for is theater.

There is a companion finding worth sitting with. METR's randomized trial with experienced open source developers found participants took 19% longer to complete tasks with AI assistance, while believing they had gone about 20% faster. The perception gap is the dangerous part. A team can feel fast, ship more diffs, and be slower end to end, because the review and rework tax is invisible in the tools everyone watches.

Shift From Reading Code to Checking Evidence

The redesign is one sentence: the pull request should arrive carrying proof, and the reviewer's job is to check whether the proof is adequate.

This is not a lower standard. It is a different one, and it is the standard every other high-volume engineering discipline settled on decades ago. Nobody inspects every weld. They inspect the weld procedure, the qualification records, and a sampling plan.

An evidence bundle for an agent-authored PR has five parts.

1. The test delta, with revert proof. Not "tests pass." Which tests are new, and which of them fail when the production change is reverted while the test stays. An agent that writes a test asserting the behavior it just wrote has proven nothing. A test that fails on revert has isolated the behavioral change.

2. Mutation coverage on the touched paths. Agents are extremely good at producing tests that execute code without asserting anything meaningful. Line coverage cannot see this. Mutation testing can: tools like Stryker or PIT introduce deliberate faults and check whether the suite catches them. Run it scoped to the diff, not the whole repo, so it finishes in CI time. A surviving mutant on a changed line is a review blocker.

3. Diff-scoped rationale. Two or three paragraphs answering what problem this solves, what approach was chosen, what alternative was rejected and why, and what the agent was uncertain about. Require the uncertainty section and reject bundles where it is empty. The single highest-value artifact we see is an agent honestly writing "I could not determine whether this cache is invalidated on tenant deletion."

4. Blast radius map. Machine-generated: which callers touch the changed symbols, which database tables are read or written, which external surfaces (public API, queue contract, feature flag) are affected, and whether the change is reversible without a data fix. This is what routes the PR to a tier.

5. Provenance. Which agent, which model version, which harness config, which tools it could reach. When a bad pattern shows up three months later, provenance is how you find every other diff produced by the same broken prompt instead of auditing the whole repo. This is the same discipline we argued for in treating harnesses as the real artifact: the thing you version and fix is the generator, not the individual output.

The reviewer's checklist becomes: does the test delta cover the claimed behavior change, do any mutants survive, does the rationale match what the diff does, and does the blast radius match the tier this PR was routed to. That is a five minute job on a 600 line diff. Reading the diff is a 90 minute job that finds less.

Tier the Merge Path

Not every change deserves the same gate. Most teams apply one policy to everything, which means the policy is calibrated for the median change and is simultaneously too heavy for trivia and too light for the changes that can take down a business.

Two rules make this hold up. First, tier assignment is computed from the blast radius map, not chosen by the author or the agent. An agent that can nominate its own change as Tier 0 will eventually do so for a migration. Second, Tier 2 is a hard floor, not a suggestion. If your policy engine cannot classify a diff, it goes to Tier 2.

For the security surfaces in Tier 2, anchor the review to a written standard rather than reviewer instinct. OWASP ASVS gives you a checklist that a reviewer can apply consistently at 9pm on a Thursday, which is more than can be said for tribal knowledge.

Google's code review guidance still supplies the right north star for what a human should be judging: whether the change improves overall code health. That is a question about design and direction. It is precisely the question an agent cannot answer about its own work, and precisely the one that gets skipped when a reviewer is grinding through line 400 of 600.

Instrument the Thing You Are Afraid Of

Every team redesigning review says the same reassuring thing: "we still review everything carefully." Measure it.

The most useful metric we deploy is rubber-stamp rate: the share of approvals issued in under 60 seconds on diffs larger than 200 lines. It requires no self-reporting and no survey. Most teams that run it for the first time find a number between 15% and 40%, and the distribution is concentrated in two or three reviewers who are drowning.

Track alongside it:

• Reviewer utilization per person, weekly. Anything sustained above 70% is a queue about to blow up.
• p50 and p90 time in review, split by tier. If p90 is 15x p50, you have a fat tail of PRs nobody wants to own.
• Revert rate within 7 days and change failure rate, split by whether the PR was agent-authored. This is the honest scoreboard. If agent PRs revert at three times the human rate, your evidence bundle is not covering the right things.
• Escaped defect rate by tier. Tier 0 escapes mean your auto-merge rules are wrong. Tier 1 escapes mean the evidence bundle is missing a check.
• Mutation score trend on changed lines. Agent-generated tests degrade this quietly over months.

One caution on volume metrics. Code churn and duplication both climb when agents write more of the code, a pattern GitClear has tracked across large public repositories. Lines added is now close to meaningless as a signal. Reverts and escaped defects are what you have left.

Getting There Without Stopping Delivery

A 90 day sequence that works:

Weeks 1 to 3, measure only. Pull PR telemetry from your forge. Compute the five metrics above on the last six months. Do not change policy yet. You need a baseline, and you need the rubber-stamp number to make the case internally.

Weeks 4 to 7, build the bundle. Add revert-proof test checking and diff-scoped mutation testing to CI. Make the agent harness emit rationale, uncertainty, and provenance into the PR body. This is harness work, not process work, and it is where the progression from assisted to agentic engineering actually gets built.

Weeks 8 to 9, classify. Write the blast radius analyzer and run it in shadow mode. Compare its tier assignments against what humans would have chosen on the last 200 PRs. Fix the disagreements before it gates anything.

Weeks 10 to 13, run both paths. Route half of eligible PRs through evidence review and half through the old process. Compare escaped defects and cycle time. Then switch the default and keep the old path available as an override.

The failure mode to avoid: adopting an automated review bot, seeing comment volume rise, and declaring the problem solved. Automated reviewers are good at the class of defect that is visible in the diff and bad at the class that comes from the change existing at all. They belong in Tier 0 and Tier 1 as noise reduction. They do not substitute for a person deciding whether a schema change should ship. The same self-checking dynamic we described in building testing harnesses that improve themselves applies here: an evaluator inside the same loop as the generator inherits its blind spots.

How OpenNash Can Help

Most of the work in this post is not model work. It is plumbing: telemetry on an existing PR queue, a blast radius analyzer that understands your actual service boundaries, mutation testing scoped tightly enough to fit in CI, and a harness that makes agents produce evidence instead of just diffs.

That maps directly to how we run engagements. The audit phase pulls your PR history and produces the baseline numbers, including the rubber-stamp rate nobody wants to see. Design sets the tier boundaries against your real risk surfaces rather than a generic template, and defines where human approval is mandatory. Build ships the analyzer, the CI gates, and the harness changes. Deploy hands the whole thing over with documentation and CI integration, owned by you.

We take a limited number of clients at a time and the work is senior-led, because tier boundaries are a judgment call that goes badly when it is delegated. If your review queue is growing faster than your team, book a call and we will map this to your merge path.

Be clear-eyed about who should not do this. If you merge fewer than 30 PRs a week, your queue is not the constraint and this is overhead. If your agents are still in the suggestion phase rather than opening PRs unattended, tighten your test suite first and revisit in two quarters. And if you have a strong existing review culture with small diffs and fast turnaround, borrow the evidence bundle and skip the tiering.

The Uncomfortable Version

The teams handling this well have accepted something their process documents have not caught up to: for most changes, no human is going to read the code, now or ever. Pretending otherwise produces the worst outcome, which is a review step that costs full price in cycle time and delivers a fraction of the assurance it claims.

Pick the honest version. Decide which 10% of changes get real human attention, make that attention genuinely good, and build machinery that earns the right to skip the other 90%. Then measure escaped defects and find out if you were right.