How to Compare AI Code Reviewers: GitHub's ReviewBench Method

One score cannot tell you which AI code reviewer to use. Precision measures how much noise a reviewer makes. Recall measures how many real issues it catches. GitHub's ReviewBench scores reviewers on both, splits the results by severity and category, and lets you weight the trade-off with an F-beta score. You can use the same method on your own pull requests without the benchmark.

The Key Numbers

All numbers in this table come from GitHub's ReviewBench announcement (October 5, 2026, by Michelle Zhou and Alejandro Carderera de Diego).

Why One Score Is Not Enough

GitHub says that the best review experience depends on the developer: "Some developers may want to focus only on critical issues, while others also value lower-severity, non-breaking findings. Some prefer broader coverage, while others prioritize precision and minimal noise." (GitHub)

ReviewBench uses these definitions:

• Precision: "Of the issues a reviewer surfaces, the proportion that are valid."
• Recall: "Of the known valid issues, the proportion the reviewer finds."
• Golden set: "A validated collection of known findings for each pull request."
• F-beta: a variation of F1 that puts more weight on precision or on recall.

A reviewer with high recall and low precision finds many bugs, but it also posts many comments that developers must read and dismiss. A reviewer with high precision and low recall is quiet and trusted, but it misses issues. These two reviewers can have the same F1 score. Your team still prefers one of them.

Cloudflare saw the noise problem when it started. Its first, simple approach gave "a flood of vague suggestions, hallucinated syntax errors, and helpful advice to 'consider adding error handling' on functions that already had it." (Cloudflare) If developers learn to ignore a reviewer, its recall stops mattering.

How ReviewBench Builds Its Test Set

GitHub builds the benchmark in a three-stage process: gather, deduplicate, validate. Before that, it picks which pull requests to include.

The categories include correctness, security, reliability, maintainability, and testing. GitHub's published list continues past these five.

GitHub uses Claude Sonnet 5 as the LLM grader, and it publishes the rubric and the judge. To check the judge, GitHub asked senior engineers who did not help build the dataset to label findings. "Their true/false-positive judgments agreed with ReviewBench 96.6% of the time." GitHub also versions the dataset, the judge, and the matcher for every evaluation. The matcher is the component that decides if a reviewer's finding and a golden finding describe the same issue.

Grounded and Augmented Metrics

A golden set is never complete. A reviewer can find a real bug that no source found. ReviewBench handles this with two families of metrics:

• Grounded precision, recall, and F1 "use only the existing gold-set labels."
• Augmented precision, recall, and F1 "also evaluate findings that do not match anything in the golden set." The judge decides if each unmatched finding is a true or a false positive.

GitHub uses "grounded recall as the headline cross-system comparison and augmented metrics as an additional per-system diagnostic." The reason is the denominator. Augmented recall adds each reviewer's own extra true findings to the set of known issues, so the denominator is different for each reviewer. Two augmented recall scores do not measure the same thing.

For an internal benchmark, use the same split. Compare reviewers on grounded numbers. Then read the unmatched findings of each reviewer to learn whether its false positives are real noise or gaps in your golden set.

Choose Beta From Your Tolerance for Noise

F-beta is the standard formula:

F-beta = (1 + β²) × P × R / (β² × P + R)

P is precision and R is recall. When beta is 1, the result is F1. When beta is below 1, precision has more weight. When beta is above 1, recall has more weight. ReviewBench lets users change beta, and the leaderboard re-ranks.

The table below gives our recommendation. Choose beta before you look at the results. If you choose it after, you will choose the beta that makes your favorite reviewer win.

A Worked Example: How Beta Changes the Winner

This hypothetical example uses a golden set of 5 findings across 3 PRs. Two reviewers run against it.

1. Reviewer A (quiet) posts 3 findings. 2 match the golden set. Precision = 2/3 = 0.667. Recall = 2/5 = 0.400.
2. Reviewer B (broad) posts 8 findings. 4 match the golden set. Precision = 4/8 = 0.500. Recall = 4/5 = 0.800.
3. At β = 0.5, A scores 0.588 and B scores 0.541. A wins.
4. At β = 1, A scores 0.500 and B scores 0.615. B wins.
5. At β = 2, A scores 0.435 and B scores 0.714. B wins by more.

Both reviewers find all critical issues in this set. The difference is in medium and low issues, which is why you should slice by severity before you decide.

Code: Score a Reviewer Against a Golden Set

The script below matches findings, computes precision, recall, and F-beta, and slices recall by severity. It uses only the Python standard library. The matching rule (same PR, file, and category, with a line within 3 lines) is a simple rule that we chose for this example. ReviewBench uses a semantic matcher, and this code does not copy it.

from collections import defaultdict
from dataclasses import dataclass


@dataclass(frozen=True)
class Finding:
    pr: str
    file: str
    line: int
    category: str
    severity: str


def match(predicted, golden, line_window=3):
    """Greedy one-to-one match. Returns (predicted, golden) pairs."""
    unmatched_gold = list(golden)
    pairs = []
    for p in predicted:
        best = None
        for g in unmatched_gold:
            if (p.pr, p.file, p.category) != (g.pr, g.file, g.category):
                continue
            dist = abs(p.line - g.line)
            if dist <= line_window and (best is None or dist < abs(p.line - best.line)):
                best = g
        if best is not None:
            pairs.append((p, best))
            unmatched_gold.remove(best)
    return pairs


def f_beta(precision, recall, beta):
    if precision == 0 and recall == 0:
        return 0.0
    b2 = beta ** 2
    return (1 + b2) * precision * recall / (b2 * precision + recall)


def grounded_scores(predicted, golden, line_window=3):
    pairs = match(predicted, golden, line_window)
    tp = len(pairs)
    precision = tp / len(predicted) if predicted else 0.0
    recall = tp / len(golden) if golden else 0.0
    return precision, recall, pairs


def recall_by(golden, pairs, key):
    total, found = defaultdict(int), defaultdict(int)
    for g in golden:
        total[getattr(g, key)] += 1
    for _, g in pairs:
        found[getattr(g, key)] += 1
    return {k: round(found[k] / total[k], 3) for k in total}


golden = [
    Finding("pr1", "auth.py", 42, "security", "critical"),
    Finding("pr1", "auth.py", 88, "correctness", "medium"),
    Finding("pr2", "cache.go", 10, "reliability", "critical"),
    Finding("pr2", "cache.go", 55, "maintainability", "low"),
    Finding("pr3", "api.ts", 7, "testing", "low"),
]

reviewer_a = [
    Finding("pr1", "auth.py", 43, "security", "critical"),
    Finding("pr2", "cache.go", 10, "reliability", "critical"),
    Finding("pr3", "api.ts", 30, "correctness", "medium"),
]

reviewer_b = [
    Finding("pr1", "auth.py", 41, "security", "critical"),
    Finding("pr1", "auth.py", 88, "correctness", "medium"),
    Finding("pr2", "cache.go", 12, "reliability", "critical"),
    Finding("pr2", "cache.go", 55, "maintainability", "low"),
    Finding("pr2", "cache.go", 70, "maintainability", "low"),
    Finding("pr3", "api.ts", 2, "maintainability", "low"),
    Finding("pr3", "api.ts", 19, "documentation", "low"),
    Finding("pr1", "auth.py", 120, "maintainability", "low"),
]

for name, preds in [("A (quiet)", reviewer_a), ("B (broad)", reviewer_b)]:
    p, r, pairs = grounded_scores(preds, golden)
    fs = " ".join(f"F{b:g}={f_beta(p, r, b):.3f}" for b in (0.5, 1, 2))
    print(f"{name}  P={p:.3f} R={r:.3f} {fs}")
    print(f"  recall by severity: {recall_by(golden, pairs, 'severity')}")

Output:

A (quiet)  P=0.667 R=0.400 F0.5=0.588 F1=0.500 F2=0.435
  recall by severity: {'critical': 1.0, 'medium': 0.0, 'low': 0.0}
B (broad)  P=0.500 R=0.800 F0.5=0.541 F1=0.615 F2=0.714
  recall by severity: {'critical': 1.0, 'medium': 1.0, 'low': 0.5}

This code counts every unmatched finding as a false positive. That makes the result a grounded precision, which can be lower than the true precision. Before you reject a reviewer, read its unmatched findings. If some of them are real issues, add them to the golden set for the next version. Change the version number each time, as GitHub does.

Build a Small Internal Benchmark

Use these steps to compare reviewers on your own code. They follow the ReviewBench method at a smaller scale.

1. Export merged PRs from the repositories that the reviewer will cover.
2. Sample PRs so that languages and repository sizes match your real mix.
3. Remove most tiny single-file changes. Keep medium and large PRs.
4. Split the sample into a small dev set and a full set. GitHub iterates on a 25-PR test set.
5. Collect human review comments that led to a code change.
6. Infer issues from follow-up commits that the author made after review.
7. Run your static analysis tools and record their findings.
8. Run two or more models from different families as extra finding sources.
9. Merge findings that describe the same underlying issue.
10. Label each finding as true, relevant, and non-trivial, or reject it.
11. Add a severity and a category to each accepted finding.
12. Ask two engineers to label a sample independently. If they often disagree with the judge, fix the rubric first.
13. Choose beta before you run any reviewer.
14. Run each reviewer more than once. GitHub runs the full set in three rounds.
15. Compare reviewers on grounded precision, recall, and F-beta, sliced by severity.

Step 12 checks your judge the same way GitHub checked its own: against engineers who did not build the dataset. Step 14 matters because small differences between reviewers can be smaller than the run-to-run variation. Our guide to leaderboard gaps smaller than error bars shows how to check this. For more on why generic benchmarks do not prove your agent works, see Workflow-Specific Evals.

Check Offline Results Against Production

An offline benchmark is useful only if its results agree with production. GitHub uses the addressed rate as the production counterpart of precision. It is "the percentage of CCR comments that an LLM determines prompted a developer to make a corresponding code change." CCR means Copilot code review. For recall, the production counterpart is how much more human review is still necessary.

GitHub reports: "Across experiments evaluated with ReviewBench before A/B testing, offline changes have consistently pointed in the same direction as what we see later in production." It gives one worked example. In a lite-tier experiment, GitHub tested a multi-model ensemble that combines several independent model runs into a single review. Relative to the production control, GitHub reported these changes:

The ensemble posted 61% more comments, but a larger share of them led to code changes. If your team has a low tolerance for noise, look at comment volume as well as the addressed rate. A reviewer can improve both precision and recall and still add 61% more comments for developers to read.

The size of a change can differ between offline and online results. For critical comments, ReviewBench predicted an increase of 227%. Production showed 262%.

So use offline results to decide which change to test, and use an online test to measure how large the effect is. Our post on harness A/B testing for agents explains how to set up that test.

How GitHub and Cloudflare Control Noise

GitHub measures noise with a benchmark. Cloudflare reduces noise in the reviewer's design. The two methods work together.

(Cloudflare)

If you build a reviewer with a coordinator like Cloudflare's, a benchmark like ReviewBench tells you if the coordinator removes noise without also removing true findings. Measure precision and recall before and after you add the coordinator.

Limits of ReviewBench

• GitHub is not neutral. GitHub sells Copilot code review, built ReviewBench, and uses it to tune the product. GitHub publishes the dataset, rubric, judge prompt, and a self-serve runner, so you can check its work.
• The sample is small. 219 pull requests is enough for a benchmark, but per-category slices contain few findings. Small slices give unstable scores.
• The grader is an LLM. The 96.6% agreement with senior engineers is strong, but the other 3.4% of judgments disagree. That error can be larger than the gap between two reviewers.
• Augmented recall is not comparable across systems. Use grounded recall to rank reviewers.
• Offline results predict direction, not size. In the critical-comment example, the prediction (227%) and the production result (262%) differ by 35 percentage points.
• Leaderboard scores can only go up. A submitted score is published only if it is the agent's first entry or if it beats the agent's current score.
• PR size is not real-world. Only language and repository size mirror GitHub. PR size is weighted toward the reviewable middle and tail, so there are fewer tiny changes.

When This Method Is the Wrong Choice

• If you only need style checks, use a linter. A deterministic tool has no precision problem for rules it can express.
• If your team merges few PRs, a benchmark costs more than it returns. Run two reviewers side by side for a few weeks and compare the addressed rate.
• If review is the bottleneck, a better reviewer may not fix it. Rachel Laycock argues that "maybe it's that we've been using code review to solve the wrong problems." (martinfowler.com) She cites Brian Houck, who reports that at Meta, lines of code per human-landed diff reportedly increased 106% in a year, and that DX data shows median PR size up 64%. If changes grow that fast, change the merge path itself. Our guide to reviewing AI-generated code at scale covers that redesign.

Choose an AI Code Reviewer With Evidence From Your Own Code

The right code reviewer finds the issues that cost you money and stays quiet enough that developers still read it. OpenNash puts forward-deployed engineers on your team, and we work with Fortune 2000 companies. A typical engagement for PR review automation follows these steps:

1. Build a golden set from your own pull requests, review history, and static analysis.
2. Agree on beta and on the severity slices that matter to your business.
3. Score candidate reviewers and models, then pick the right model for each type of check.
4. Ship the reviewer to a subset of repositories with cost controls, and track the addressed rate.
5. Train your engineers to maintain the golden set and to run the benchmark on each change.

At the end, you own the benchmark, the golden set, the reviewer, and the dashboards.

Book a call with an OpenNash forward-deployed engineer or email us at hello@opennash.com. In the first session, we review your top AI use cases and your data, and we show which of them give the highest business value and the fastest ROI.