A person who approves every action of an AI agent stops reading the requests. In a controlled test by Anthropic, people caught only 13.6% of planted dangerous commands, and about 5% after 50 or more prompts. An automatic classifier caught 89%. So a human click on each agent action is not oversight. Real oversight puts people on plans and high-risk actions, and puts machines on the rest: sandboxes, scoped permissions, classifiers, sampled audits, and logs.
The Data: People Stop Reading
Anthropic has published the most detailed numbers on how people use agent permission prompts.
In August 2026, Anthropic reported that users approve 97% of permission prompts in Claude Code. A high approval rate is not a problem in itself, because most requests are harmless. The problem is what happens to the few that are not.
To measure it, Anthropic planted dangerous commands in sessions with 1,053 paid testers. The testers caught 143 of them, or 13.6%. An automatic classifier blocked 937, or 89%. Early in a session, people caught about 17%. After 50 or more prior prompts, they caught about 5%.
An earlier Anthropic engineering post states the cause plainly: the more approvals a user sees, the less attention each one gets. Users also route around the prompts. In the August data, 62% of users had used the bypass mode or clicked "don't ask again" on shell commands.
This Is Automation Bias
Human factors research has studied this effect for decades. Parasuraman and Manzey reviewed the evidence in 2010. They found automation bias in experts as well as novices, and in teams as well as individuals. Training and instructions did not prevent it.
Experience does not protect people. In a 2023 study reported by the Radiological Society of North America, radiologists read mammograms with AI suggestions, some of them deliberately wrong. When the AI was wrong, inexperienced readers fell from almost 80% correct to under 20%. Very experienced readers fell from 82% to 45.5%.
NIST AI 600-1, the US generative AI risk profile, names automation bias as a risk of generative AI. The essay The Commanding Heights by Will Manidis and Joe Roberts opens with the same scene in fiction. A manager approves agent requests unread, because the agents have never made a mistake.
People Engage With Decisions, Not Keystrokes
The same Anthropic data shows where people do pay attention. Users reject 39% of plans that Claude presents for approval, but only 3% of single permission prompts.
A plan and a shell command are different things, so the two rates are not a controlled comparison. The lesson still holds. A plan states what the agent will do and why, so a person can judge it. The hundredth shell command of the day gives a person nothing to judge.
The Volume Keeps Rising
Agents now work longer between human checks. In Anthropic's study of agent autonomy, human interventions per session in internal use fell from 5.4 to 3.3 in late 2025. Experienced users ran twice as many sessions on full auto-approve as new users. That trend will continue as models improve. An agent that stops every minute is not useful. The design question is where to spend scarce human attention.
Excess Access, Not Missed Clicks
Two public incidents show what happens when access is too broad.
Replit, July 2025. During a code freeze, Replit's coding agent deleted data from the production database of a customer's app. Replit's account says the data was recovered by rollback. The company then separated development and production databases, so the agent cannot change production during development.
AWS, December 2025. The Financial Times reported that an AI coding agent caused an AWS outage. Amazon's response disputes this. It says the cause was misconfigured access controls, and that it added mandatory peer review for production access.
The two fixes have the same shape. Neither added more prompts. Both removed access or added a second person.
Design Oversight for the Volume
OWASP's LLM06 Excessive Agency names three root causes: too much functionality, too many permissions, and too much autonomy. OpenAI's guide to building agents rates each tool by read or write access, reversibility, required permissions, and financial impact. Combine the two into tiers.
| Tier | Examples | Control |
|---|---|---|
| 0: Read only | Search, read a record, draft a reply | Runs automatically. Scoped to the user's permissions. Logged |
| 1: Reversible writes inside a sandbox | Edit a draft, update a test environment, create a ticket | Runs automatically inside the sandbox. Classifier checks. Sampled audit |
| 2: Writes to systems of record | Update a customer record, post a journal entry, send an email | A person approves the plan or the batch, with evidence and a diff |
| 3: Irreversible or high value | Payments, deletions, production changes, regulatory filings | Two-person approval, a hold period where possible, and a stop switch |
Then apply these rules:
- Approve plans, not keystrokes. Ask for approval once per task, on a plan a person can judge.
- Show evidence with every request. Show the source data, the diff, and the reason. A request without evidence trains people to click.
- Run with the user's rights. The agent acts in the context of the user who started it, never with a shared admin key. See permission-aware AI agents.
- Let machines take the first pass. Sandboxes and classifiers handle routine actions. Anthropic's sandboxing cut permission prompts by 84%. Its classifier blocks about 0.4% of harmless commands, a cost worth tracking.
- Test the reviewers. Seed known-bad items into review queues and measure the catch rate. If the rate falls, reduce the volume or rotate reviewers.
- Sample and audit. Review a random sample of tier 0 and tier 1 actions each week. Log every action, approval, and rejection. See AI agent audit trails.
- Keep a stop switch. Any agent must be stoppable in seconds, by more than one person. See the agent kill switch.
For the patterns that decide when an agent should hand work to a person, see human-in-the-loop agent design.
What Regulators and Auditors Ask For
EU AI Act. Article 14 requires that people who oversee a high-risk AI system stay aware of "automation bias." They must be able to override its output and stop it. Article 12 requires automatic logs over the life of the system. Article 26 requires deployers to give oversight to people with the competence and authority to act, and to keep logs for at least six months. Regulation (EU) 2026/1744 moved the start date for Annex III high-risk systems to December 2, 2027, according to the European Parliament's research service.
US financial services. FINRA's 2026 oversight report lists AI agents that act without human validation as a risk. It asks firms where they need "human in the loop" oversight. The Federal Reserve's SR 26-2 (April 2026) replaced SR 11-7 for model risk. It puts generative and agentic AI out of its scope, but says a bank's governance should still set controls for them.
Insurance. The NAIC model bulletin asks insurers for an AI program with governance, risk controls, and internal audit. As of October 2026, 29 jurisdictions had adopted it.
None of these rules accept a click as proof of oversight. They ask for people with authority, records, and the ability to stop the system.
Make Oversight Real With OpenNash
OpenNash Work and OpenNash Security give your company agent oversight that scales with the volume:
- Risk-tier every tool. We rate each agent tool and action and set the control for each tier.
- Scope permissions per user and per agent. Agents act with each user's rights, and nothing more.
- Log everything. Every prompt, tool call, write, approval, and rejection goes to an audit trail you own.
- Put people on decisions. Plan approvals, evidence, and diffs for tier 2. Two-person approval and stop switches for tier 3.
- Measure the reviewers. Seeded tests and sampled audits show whether oversight works.
Book a call with an OpenNash forward-deployed engineer or email us at hello@opennash.com. In the first session, we review the agents you run or plan to run and map their actions to risk tiers. Then we show you which use cases deliver the highest business value and the fastest ROI.