Verifying AI Generated Code: How Meta and Shopify Make Agents Prove Their Work

Meta and Shopify both make their coding agents prove their work before a human reviews it. The agent attaches evidence to the pull request: build results, test results, and recordings of the app running. The reviewer checks the evidence first and reads the diff closely only when something looks wrong. This post compares how each company built its verification layers, then gives a ranked list of layers to add to your own agent workflow.

What the Two Companies Published

Both posts came out in October 2026. Each one describes a coding agent that works on mobile apps and the checks around it.

Shopify does not give a time span for the 6% to 61% change. That number measures adoption, not review quality. Shopify partly credits "storytelling and internal ads" for it.

Why "It Compiles" Is the Wrong Bar

Code that compiles can still be wrong. Shopify says it directly: "Just because the code compiles doesn't mean the feature behaves as expected."

Meta explains what happens when the checks are weak. The work does not get faster. The bottleneck moves. In Meta's words, "Weak signals don't remove the constraint; they move it from writing code to reviewing code that looks finished and isn't."

Simon Willison makes the same point from the individual engineer's side. When he runs many agents in parallel, "the natural bottleneck on all of this is how fast I can review the results."

Agent evaluation research gives the general rule. Grade the outcome, not the agent's claim. Anthropic's example: a flight-booking agent can say the flight is booked, but "the outcome is whether a reservation exists in the environment's SQL database." A coding agent that says "fixed" is the same case. The evidence is the build, the tests, and the app running.

We covered this idea in general form in Verifiable Outcomes: Why the Best AI Agents Start With Boring Proof. Meta and Shopify show what it looks like for mobile code.

How Meta Built Verification Into the Forum Agent Loop

Meta set a clear definition of done. A task is complete only after automated tests and visual checks pass. Meta calls this setting "the bar at verification rather than compilation."

The loop

The agent runs a loop that Meta describes as "plan, implement, build, test, correct, repeat." Meta put three automated pass/fail signals inside that loop:

• Build status. The app must build.
• Unit tests. The tests must pass.
• UI spec matching. The screen is checked against a UI spec. Meta does not describe in detail what the visual checks compare against.

Because the signals run inside the loop, the agent fixes its own failures before a human sees the work. The output is a pull request with a pre-populated test plan and recorded verification sessions.

Navigation agents

A navigation agent is an agent that uses the running app the way a person does. Meta deployed them to "interact with the live app exactly like a person would". They perform actions, read the screen, and report regressions. Meta also uses them for stress tests: aggressive scrolling, random group browsing, rapid navigation, and repeated media loading.

To help agents see the app, Meta built tools that read the live view hierarchy from the simulator, list the elements on screen, and look up the path to a target view.

Hooks and rules

Meta adds checks at fixed points in the agent's work. A hook is a check that runs automatically at a set moment. Meta's examples are a linter after every agent turn, a simulator connectivity check before install, and test validation before a pull request is created.

Rules handle repeated mistakes. "When the agent repeats a mistake more than twice, we encode the correction once, for everyone." This turns a reviewer's comment into a permanent check.

Human on the loop

Meta describes its target as human on the loop. The agent runs the iteration. The engineer sets the goal, watches the run, steps in when it drifts, and approves the result. Meta calls this "the target we are driving toward, not a finished state."

The result, according to Meta, is that "what reaches review is a smaller set of higher-confidence results rather than a pile of output to sift through."

How Shopify Gave River a Mobile Dev Environment

River is Shopify's AI agent in Slack. We described why it exists in Internal AI Workspace: Why Ramp Glass and Shopify River Exist. This post focuses on how River proves its mobile changes.

The problem: an agent that could code but not check

River ran on Linux. iOS builds need a Mac. Shopify says River "always stated that an iOS build cannot run on Linux, so it could code but not verify the results." Engineers could not run many agent workflows locally either. Three or four workflows were enough to max out a laptop.

The fix: one remote environment per session

Shopify built Remote Dev on a fleet of remote Macs, connected over SSH. Android runs on a separate Linux VM with nested virtualization. Each session follows the same setup:

1. Claim a Mac.
2. Synchronize the branch.
3. Start a prebuilt Android VM and attach it.

Setup first took over 30 minutes. After Shopify added a warm pool of ready machines, it went down to a few minutes.

Teaching the agent to verify

Infrastructure alone was not enough. Shopify created a skill to "teach agents how to test their changes and prove their work." A skill is a packaged procedure that the agent loads when it needs it. With the skill, the agent can implement changes, build the app, run tests, interact with the simulators, collect screenshots, and create a pull request. If something fails, it reads the logs, fixes the issue, and tries again.

Early versions had problems. River forgot about the remote Mac, stayed silent during long remote steps, or took a long time to reply. Shopify moved long tasks into tmux panes that River can watch in real time, and wrapped the SSH details in a CLI.

Before-and-after recordings and Live Preview

For bug fixes, River reproduces the issue, records the broken behavior, implements the fix, and repeats the same steps. Shopify says, "Instead of just looking at the code, we can actually see what changed and verify the result."

Live Preview streams the iOS Simulator and Android Emulator to a web page over WebSockets. Anyone with the link can use the running app in real time. Every Remote Dev session gets its own Live Preview for both platforms.

Note that these are simulators and emulators on remote machines, not physical phones.

Verification Layers Side by Side

This table compares the verification layers in the two posts. "Not stated" means the post does not mention it. It does not mean the company does not do it.

The pattern is the same in both columns. Each layer produces an artifact that a reviewer can check in less time than reading the diff.

From Task to Reviewed Pull Request: Evidence at Each Step

This table merges the two pipelines into one sequence. The right column shows the evidence each step should leave for the reviewer.

Steps 4 to 7 repeat until every check passes. That repetition happens before a human spends any time on the change.

A Worked Example: Reviewing One Agent Bug Fix

This is a hypothetical example. It shows how a reviewer uses the evidence from the table above.

1. A ticket says the "Join group" button does nothing on Android.
2. The agent reproduces the bug in the emulator and attaches a 10-second "before" recording.
3. The agent changes one click handler. The build passes. The existing tests pass, and the agent adds one new test.
4. The agent replays the same taps and attaches an "after" recording that shows the join confirmation.
5. The reviewer watches both recordings. The behavior matches the ticket.
6. The reviewer reads the diff for scope. It touches one file, as expected.
7. The reviewer opens the Live Preview link and taps the button once on iOS to check the other platform.
8. The reviewer approves.

If the "after" recording showed a second, unrelated screen change, the reviewer would read the diff line by line. The evidence tells the reviewer where to spend attention.

What the Sources Do Not Prove

Keep three limits in mind before you copy this pattern.

• No review-time numbers. Meta calls sign-off "fast and well-evidenced." Shopify calls review "much easier." Neither publishes a measurement. Do not promise your team a percentage.
• The verifiers set the ceiling. Meta writes that "how much we can safely hand to the loop is gated by how much we can trust the verifiers." A flaky test or a weak UI check gives false confidence.
• Humans still approve. Both companies keep a human sign-off on every change.

Datadog takes a stronger position. Its harness-first approach says, "instead of reading every line of agent-generated code, invest in automated checks that can tell us with high confidence, in seconds, whether the code is correct." Datadog argues that a harness can replace code review as the main source of correctness. Meta and Shopify do not go that far. Treat Datadog's view as the far end of the range, not as the shared view.

For the merge-path side of this problem, see Reviewing AI Generated Code at Scale: Redesigning the Merge Path.

Which Verification Layers to Add First

This is our recommended order. It puts cheap, reliable checks first and expensive, harder-to-trust checks later. Each layer lists the evidence it should attach to the pull request.

Two conditions change this order. If your product has no UI, skip layers 4 to 6 and 8. Put the effort into tests and output checks against real system state. If your tests are flaky, fix them before you add layer 3 or above. A flaky layer teaches reviewers to ignore the evidence.

Navigation agents are a form of simulation testing. Our guide to Agent Simulation Testing covers how to build a simulated user and score its runs.

An Evidence Check for Agent Pull Requests

You can enforce the evidence rule in CI. This script fails a pull request when its description lacks the required sections. Run it as a required check on pull requests that an agent opens. Pass the PR description in a file. The section names and the ui-change and bugfix labels are our own conventions, not taken from Meta or Shopify. Change them to match your repository.

#!/usr/bin/env python3
"""Fail an agent pull request that does not attach verification evidence."""
import re
import sys

REQUIRED = {
    "Build": r"^##\s*Build\b",
    "Tests": r"^##\s*Tests\b",
    "Test plan": r"^##\s*Test plan\b",
}
UI_LABEL = "ui-change"
BUGFIX_LABEL = "bugfix"


def has_link(section_text: str) -> bool:
    return bool(re.search(r"https?://\S+", section_text))


def section(body: str, heading: str) -> str:
    match = re.search(rf"^##\s*{heading}\b(.*?)(?=^##\s|\Z)", body, re.M | re.S)
    return match.group(1) if match else ""


def main(body_path: str, labels: list[str]) -> int:
    body = open(body_path, encoding="utf-8").read()
    missing = [name for name, pattern in REQUIRED.items()
               if not re.search(pattern, body, re.M)]

    if UI_LABEL in labels and not has_link(section(body, "Screenshots")):
        missing.append("Screenshots (with links)")
    if BUGFIX_LABEL in labels:
        recordings = section(body, "Recordings")
        if not (re.search(r"before", recordings, re.I)
                and re.search(r"after", recordings, re.I)
                and has_link(recordings)):
            missing.append("Recordings (before and after, with links)")

    if missing:
        print("Missing verification evidence: " + ", ".join(missing))
        return 1
    print("Verification evidence present.")
    return 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1], sys.argv[2:]))

This check only proves that evidence is present. It does not prove the evidence is correct. The build and test sections should link to CI runs that the agent cannot edit, so the reviewer can check the source.

When This Pattern Is the Wrong Choice

Recorded evidence costs infrastructure and time. It is not always worth it.

• Small, low-risk changes. For a copy change or a dependency bump with good tests, the build and test results are enough. Recordings add noise.
• No trusted test suite. If your tests do not catch real bugs, the evidence will look strong and mean little. Fix the tests first.
• Low agent volume. Shopify built Remote Dev because many agent sessions needed many Macs. If an agent opens a few pull requests a week, a reviewer can run the app by hand.
• Changes the checks cannot see. Security logic, data migrations, and concurrency bugs rarely show up in a screen recording. Read these diffs line by line.

Make Agent Output Something Your Team Can Approve

Agent code helps your business only when your team can approve it with confidence and ship it. OpenNash puts forward-deployed engineers on your team and works with Fortune 2000 companies to build these verification pipelines. A typical engagement follows these steps:

1. Audit your current review process and find where agent pull requests stall.
2. Put build, test, and hook checks inside the agent loop, with results attached to each pull request.
3. Add runtime evidence (screenshots, before-and-after recordings, preview links) where your product needs it.
4. Measure verifier reliability, so your team knows which checks it can trust.
5. Train your reviewers to review evidence first and the diff second.

At the end, you own the agent workflow, the verification harness, the CI checks, and the documentation.

Book a call with an OpenNash forward-deployed engineer or email us at hello@opennash.com. In the first session, we review your top AI use cases and your data, and show which ones give the highest business value and the fastest ROI.