An expense-reimbursement agent reads a receipt, fills the form, and checks the policy correctly. Then it stalls because the employee's manager changed last week and the approval directory is out of date. Did it automate the work?
Partly. That answer is less exciting than "AI employee," but it is much more useful for an enterprise buyer.
Jobs are bundles of tasks. Some tasks are repetitive and verifiable. Others depend on relationships, judgment, accountability, or exceptions hidden in a veteran employee's head. An agent can create real value on the first group without being able to replace the person responsible for the whole job.
The right planning unit is therefore the task, not the role. Score the task, assign the right level of autonomy, and measure whether work was completed correctly after review and recovery costs are included.
The Evidence Supports Task Automation, Not Digital Employees
TheAgentCompany is a useful reality check because it tested more than chat answers. The NeurIPS 2025 benchmark created a simulated software company and gave agents 175 consequential tasks across software engineering, project management, finance, human resources, and administration. Agents could browse sites, write code, operate workplace software, and communicate with simulated colleagues.
The best system in the TheAgentCompany paper completed 30.3 percent of tasks fully and scored 39.3 percent when partial credit was included. It averaged 27.2 steps and $4.20 of model cost per task, before any enterprise review or recovery cost. Because full runs were expensive, the study did not report confidence intervals; small differences between systems should not be overread. The result came from one simulated software company, a specific task set, and the models available during the study.
The durable lesson is the gap between progress and completion. An agent can do five useful steps and still leave the business task unfinished.
Examples are easy to recognize:
- It gathers reimbursement evidence but cannot resolve a missing approval.
- It drafts a project update but assigns a dependency to the wrong owner.
- It writes a code change but does not confirm that deployment succeeded.
- It finds a customer record but cannot distinguish two merged accounts.
- It prepares an onboarding checklist but misses a country-specific requirement.
Partial progress may still save time. It becomes a bad investment when reviewers must reconstruct the task, discover hidden mistakes, and repair system state. That is why enterprise evaluation must count accepted completions and correction effort, not impressive intermediate output.
Task duration matters too. Research from METR on measuring AI ability to complete long tasks uses the length of time a human expert would need as a way to describe task difficulty. Longer work tends to introduce more state, tools, decisions, and chances for recovery to fail. The exact performance curve will change quickly as models improve, but workflow length remains a practical risk signal.
Do not read the 30.3 percent figure as "agents can replace 30.3 percent of employees." Read it as evidence that capability varies at task level and that end-to-end completion is substantially harder than producing useful fragments.
Score the Task Before You Automate It
Candidate selection is where most of the return is won or lost. Teams often start with the process that executives complain about most. That can be the worst pilot if it involves six systems, unclear ownership, irreversible actions, and exceptions nobody has documented. When the process itself is disputed or mostly tacit, use process discovery to establish the real steps before assigning an automation score.
Use an eight-factor scorecard. Rate each factor from 1 to 5, where 5 means easier and safer to automate.
| Factor | Score 1 | Score 5 |
|---|---|---|
| Frequency and value | Rare, low-cost work | Frequent, meaningful cost or delay |
| End-state clarity | Success is subjective | Final state can be asserted |
| System count | Many poorly integrated systems | One or two stable systems |
| Human coordination | Negotiation or several handoffs | No coordination or one clear approval |
| Input consistency | Missing, unstructured, variable | Known fields and reliable context |
| Reversibility | Error is costly or permanent | Change can be rolled back safely |
| Permission scope | Broad or privileged access | Narrow user-scoped access |
| Exception burden | Exceptions dominate | Happy path covers most cases |
Frequency alone is not enough. A high-volume task with an ambiguous result can create high-volume review work. Ease alone is not enough either. Automating a two-minute task performed monthly will not repay the operating overhead.
The best first task often has an odd combination: boring inputs, a visible business bottleneck, and an objective final state. Think of attaching approved documents to a CRM opportunity, preparing a renewal brief from known systems, or classifying invoices into a fixed exception queue.
Define completion as state, not prose. "Agent generated a good summary" is hard to test. "Opportunity note contains the last meeting date, open support severity, renewal amount, and citations to each source" can be tested. "Invoice processed" is vague. "Invoice, purchase order, amount, vendor, and approval status match; exception record created when any field fails" is executable.
The NIST AI Risk Management Framework organizes AI risk work around govern, map, measure, and manage. Applied to task selection, that means naming the owner, mapping the workflow and affected people, measuring performance and harm, then operating controls after launch. A scorecard without an accountable process owner is just a prioritization workshop.
Also count the people who absorb failures. A system that saves sales representatives five minutes but sends confusing exceptions to finance may move cost rather than remove it. Include reviewer time, escalations, reopenings, and cleanup in the baseline.
Put Work Into Four Automation Lanes
Not every good use case needs autonomous action. Assign each task to one of four lanes based on the scorecard and its consequence.
| Lane | Best fit | Example | Required control |
|---|---|---|---|
| Automate | Bounded, reversible, objectively verifiable | Tag and route a support ticket | Postcondition check and rollback |
| Draft and approve | High value with consequential output | Prepare a customer credit memo | Named human approves before write |
| Assist | Ambiguous work where preparation helps | Research an acquisition target | Citations and editable work product |
| Keep human-led | Negotiation, ownership, sensitive judgment | Performance review or contract concession | AI may retrieve facts, not decide |
Automate means the agent can perform the task within narrow permissions and verify the result. It does not mean unlimited access. A routing agent can update the queue and priority field without receiving permission to close tickets or email customers.
Draft and approve is often the highest-return lane. The agent gathers data, checks rules, prepares the action, and presents the evidence to an authorized person. The human makes one meaningful decision instead of doing clerical assembly. Our guide to human-in-the-loop agent design describes approval patterns for actions with financial, legal, or customer impact.
Assist is right when the output remains open-ended. Research, comparison, meeting preparation, and first drafts can save hours even if a person owns the final result. Require citations and make uncertainty visible. Do not disguise assistance as full automation in the business case.
Keep human-led is a deliberate architecture choice, not a failure of ambition. Negotiation, personnel decisions, conflict resolution, and accountable sign-off depend on authority and relationships. An agent may prepare evidence or record the decision, but a named person should own it.
A common mistake is assuming every task should move rightward toward autonomy as the model improves. Some work should stay in draft-and-approve permanently because the approval is the governance act. Faster models do not remove fiduciary responsibility.
The lanes can also split a single workflow. In customer credits, the agent might autonomously collect account facts, draft the recommendation, require approval for the credit, then autonomously record the approved decision. That is safer and often faster than choosing between "manual" and "fully autonomous."
Design the Workflow Around Failure and Recovery
A demo follows the happy path. Production work is defined by what happens after an expired token, duplicate record, ambiguous request, changed policy, unavailable approver, or partial tool failure.
Before launch, write the workflow as a state machine with explicit boundaries:
- What starts the task?
- Which identity is the agent acting for?
- Which facts must be read, and from which authoritative systems?
- Which decisions belong in deterministic rules?
- Which actions may the agent take without approval?
- What proves each action succeeded?
- Which failure stops, retries, rolls back, or escalates?
- Who owns an unresolved case?
Tool responses are not proof of business completion. An API returning 200 OK may mean the request was accepted, not that the intended state exists. Read the record back. Compare expected fields. Confirm downstream job status. Detect duplicates. Save an idempotency key so a retry does not create a second refund or task.
Permissions should match the assigned lane. The OWASP guidance on excessive agency in LLM applications recommends minimizing tool functionality, permissions, and autonomy. That is a practical design rule: give the agent the smallest tool surface required for this task, not the employee's entire account.
Separate reading, drafting, approving, and writing. A research step may use broad read access but no mutation tools. A write step should receive a validated payload, a narrow credential, and a postcondition. Our agent writeback patterns cover idempotency, audit records, and verification in more detail.
Design escalation as a productive output. An agent should pass the human a concise reason, gathered evidence, attempted steps, current system state, and recommended next action. "I could not complete this" creates work. "Vendor ID is duplicated across two active records; no mutation attempted; finance owner must choose the canonical record" advances the case safely.
Good automation is not the absence of humans. It is the removal of low-value human work while preserving human authority where it matters.
Prove Value in Shadow Mode Before Granting Autonomy
Start with one task and 50 to 100 representative cases. Run the agent alongside the current process without allowing consequential writes. Compare proposed actions with what employees actually did and with the verified final state.
Measure:
- Fully correct completion rate
- Partial progress that reviewers accepted
- Human review and correction minutes
- Incorrect actions that would have occurred
- Appropriate and missed escalations
- Cost per accepted completion
- Elapsed cycle time
- Performance by common case and exception type
Partial credit is useful for product diagnosis, but the ROI model must be stricter. If an agent completes 70 percent of steps but every case still needs a ten-minute audit, the saved labor may be small. Report accepted outcomes and net time saved after review.
Use a promotion ladder:
Observe. Capture how good operators complete the task and where they diverge.
Recommend. The agent proposes the next step and evidence, but changes nothing.
Draft. The agent prepares a structured action for approval.
Act with approval. A person authorizes a specific validated change.
Limited autonomy. The agent performs only low-risk cases that meet explicit criteria.
Expand carefully. Add one exception class, tool, or permission at a time and rerun regression tests.
Anthropic's engineering guidance on building effective agents recommends starting with the simplest solution and adding agentic complexity only when it improves outcomes. That is sound operating advice. A deterministic workflow with one model step is easier to test than an open-ended agent and may solve the same business problem.
Maintain a holdout set containing high-risk and rare cases. Re-run it after model, prompt, tool, policy, or connector changes. Train operators and reviewers to label corrections consistently; our human rollout plan explains how those roles keep the system improving after launch.
Set a stop rule before the pilot. Pause expansion if severe errors exceed the threshold, reviewers cannot agree on correctness, source data is unreliable, or human correction erases the expected savings. Waiting is a valid choice when the workflow is not ready.
Build the Business Case Around Completed Work
An AI-agent budget should not be justified by messages generated, tool calls made, or tokens saved. The unit that matters is a correctly completed task.
Calculate the current baseline:
Current cost per completed task =
labor time + delay cost + rework + error cost
Then calculate the agent path:
Agent cost per completed task =
model and infrastructure cost
+ review time
+ correction and escalation time
+ expected failure cost
+ operating and maintenance cost
Include the value of faster cycle time where it is real: earlier invoice processing, faster customer response, or shorter sales preparation. Do not assign revenue to every minute saved unless the business can explain the causal link.
Evaluate platforms fairly. A packaged product is usually right for a standardized process with common systems, mature controls, and little company-specific logic. Custom implementation is justified when the work crosses proprietary systems, depends on unique rules, or needs tight ownership of permissions, traces, and handoffs. A company with undocumented steps and no stable definition of success should wait and conduct process discovery first.
The first portfolio should mix lanes. One autonomous low-risk task proves operations, one draft-and-approve task proves consequential value, and one assistive task tests adoption. That produces a more honest capability map than a single flashy pilot.
How OpenNash Can Help
OpenNash helps enterprises identify the tasks worth automating, then build the smallest controlled workflow that can prove value. We map operator steps, score candidates, define completion assertions, separate deterministic rules from model judgment, and design permissions, approvals, traces, and recovery before production access is granted.
The engagement is measured against completed work: acceptance rate, reviewer effort, cycle time, cost, and failure severity. The client owns the workflow code, evaluation suite, runbook, and system integrations after handoff.
Book a call to score one team's recurring tasks and choose what to automate, what to draft, and what should remain human-led.