Production Evals for AI Agents: Measuring Workflow Behavior

A support automation demo can feel almost finished after twenty minutes. The agent greets the customer, retrieves a help article, calls an order API, drafts a reply, and closes with a polite handoff. The pilot gets funded.

Then production arrives.

One customer asks for a refund and a shipping correction in the same thread. Another has a stale address in the CRM. A third is eligible for a credit only if the agent reads a policy update from last week. A fourth hits a backend timeout and risks creating the same ticket twice. The model did not suddenly get worse. The demo measured the answer. Production measures the workflow.

Nishant Gupta makes this point in Production Evals For Agentic AI Systems: for agents, the evaluation question is no longer "did the model produce the right answer?" It is "did the system behave correctly?"

That shift sounds small until an agent touches real operations. Agents plan, retrieve, call tools, write records, retry after failures, and decide when to ask for help. They are workflow participants. A mid-market company cannot manage that with benchmark scores and a thumbs-up dashboard.

Production evals turn agent behavior from a demo into something measurable enough to operate.

Benchmarks Are Not Release Gates

Benchmarks still matter. They tell you whether a model can reason, code, classify, or follow instructions. They are useful when choosing a starting model or tracking broad progress.

They do not tell you whether an accounts receivable agent should update NetSuite, whether a support agent will respect a new refund threshold, or whether a sales operations agent will avoid overwriting a manually corrected CRM field.

OpenAI's Evals documentation frames evals as a way to test LLMs and LLM systems, including custom evals for the use cases a team cares about. The important word for operators is "systems." The model is one part of the system. The production workflow also includes:

• The prompt and policy text.
• Retrieval and memory.
• Tool schemas and permissions.
• API behavior, including timeouts and partial failures.
• Routing rules and human approval gates.
• The final action taken in a system of record.
• The trace that proves what happened.

If you change any one of those pieces, behavior can change. A stronger model can call a dangerous tool more confidently. A clearer prompt can improve one queue and hurt another. A new retrieval index can surface the wrong policy version. A cheaper model route can save money while quietly increasing escalations.

For buyers, a benchmark score is not a release decision. A release decision needs a workflow-specific eval suite with a threshold the business understands.

The Eval Unit Is a Workflow Case

The useful unit is not "prompt number 42." It is a case from the workflow.

For customer support, a case might include the customer message, account state, order history, policy snippets, allowed tools, expected tool calls, and correct resolution. For finance, it might include an invoice, purchase order, receipt, vendor record, approval limit, and expected exception decision. For compliance, it might include a contract folder, clause checklist, reviewer rubric, and required citations.

LangSmith's evaluation concepts make the same split that production teams need: offline evals use curated datasets before deployment, while online evals monitor production traces after launch. The offline set gives you a release gate. The online stream tells you what reality did that your test set missed.

A good workflow case has six parts:

1. Input: The actual request and state the agent must handle.
2. Allowed actions: The tools, systems, and permissions available for the case.
3. Expected behavior: The outcome, escalation, refusal, or write-back that should happen.
4. Assertions: Hard checks for facts, API calls, policy limits, required fields, and forbidden actions.
5. Judgment review: Human or model grading for tone, completeness, ambiguity, and business taste.
6. Trace fields: The evidence needed to replay the run and explain the result.

This is where many pilots stay too vague. "The agent should answer accurately" is not an eval. "For refund requests above $500, the agent must not issue the refund tool call and must create an approval task with policy citation R-17" is an eval.

The best cases come from real operations. Pull resolved tickets, invoices, quote requests, onboarding emails, renewal notes, and exception logs. Include boring cases because volume lives there. Include edge cases because risk lives there. Include past failures because they are free tests your users already wrote.

What Production Evals Actually Measure

For an agent, final answer accuracy is only one row in the scorecard. The eval should measure the stages where production breaks.

This is the operational version of the classic SRE lesson. Google's SRE book points teams toward signals that compare over time, alert on breakage, and debug after the fact. Agent systems need the same discipline, plus behavior metrics for tool use, policy, recovery, and human handoffs.

For agents, an "error" is not just a 500 response. It can be an HTTP 200 with the wrong action, a correct answer sent to the wrong customer segment, a summary that skipped approval, or a fast response that created expensive rework.

That is why production evals should report multiple rates:

• Task completion rate.
• Policy violation rate.
• Required tool call success rate.
• Forbidden tool call rate.
• Retrieval freshness rate.
• Escalation precision and recall.
• Duplicate action rate.
• Human edit distance.
• Cost per resolved case.
• Latency by case type and failure type.

The dashboard should make one thing plain: is the agent getting safer and more useful inside the workflow, or just producing more activity?

Traces Are the Raw Material

You cannot evaluate behavior you did not capture.

OpenAI's Agents SDK tracing docs describe traces and spans for agent runs, including generation spans and function spans that can capture model inputs, outputs, and function call data. The docs also flag the privacy point: sensitive data capture must be controlled. That is exactly the tradeoff operators need to manage. You need enough detail to debug and score the workflow, without spraying customer data through every analytics tool in the company.

At minimum, each production trace should capture:

• User input and normalized intent.
• Prompt, model, and route version.
• Retrieved documents and versions.
• Tool calls, arguments, outputs, errors, and retries.
• Permission checks and approval gates.
• Final response or system action.
• Latency, cost, reviewer edits, and business outcome.

Without those fields, the team is stuck arguing about anecdotes. With them, a failed run can become a test, and a policy owner can see whether the agent followed the rule as written.

The trace should not end at the chatbot response. If the agent opened a ticket, updated a CRM record, drafted a quote, or wrote to an ERP, the trace needs the downstream action and result. The business cares about the work, not the sentence.

Offline and Online Evals Need Each Other

Offline evals are your pre-release gate. They run a fixed set of cases against a new prompt, model, tool schema, retrieval index, or workflow change. They tell you whether the change made known cases better or worse.

Online evals watch live traffic. They catch new input shapes, policy drift, data freshness issues, vendor API changes, and cases missing from the first dataset. They do not always have reference answers, so they rely on heuristics, safety checks, reviewer sampling, and business outcomes.

The loop matters more than the label:

1. Run offline evals before release.
2. Ship only if the workflow clears the threshold.
3. Monitor live traces after release.
4. Review failures and human edits weekly.
5. Promote useful failures into offline cases.
6. Repeat the same release gate on the next change.

That is the bridge between this post and our earlier field guide, Every Production Failure Should Become an Eval Case. A production failure that does not become a test is just a recurring meeting waiting to happen.

It also connects to Loops and Evals: How You Know AI Agents Work in 2026: the agent loop performs the work, the verification loop scores it, and the improvement loop turns traces into better cases, prompts, tools, and policies.

The Release Gate Should Be Business-Readable

The eval suite cannot live only in engineering language. If an operating partner, CFO, support leader, or compliance owner cannot understand the gate, the system will ship on trust or get blocked by fear.

Define release thresholds in business terms. The NIST AI Risk Management Framework is useful here because it treats evaluation and risk management as part of the AI system lifecycle, not an afterthought.

• No P0 policy violations in the golden set.
• At least 95% pass rate on Tier 1 support cases.
• At least 90% correct escalation on ambiguous refund cases.
• No duplicate write actions under retry tests.
• P95 latency below the support SLA for standard cases.
• Cost per resolved case below the planned budget.
• Human edit distance no worse than the current baseline.

These thresholds do not need to be perfect on day one. They need to be explicit. A team can ship a low-risk drafting assistant at 85% because every output is reviewed. The same team should not ship an autonomous refund tool at 85% if the failing 15% includes policy breaches.

The gate should also be sliced by failure type. A single aggregate score hides the cases executives care about. A 92% pass rate sounds good until the 8% are all high-dollar exceptions. Break the score down by intent, tool path, policy area, model route, and risk tier.

What Mid-Market Operators Should Build First

Do not start with a platform bake-off. Start with one queue where the business already knows what good work looks like: support ticket triage, invoice exception handling, renewal prep, quote cleanup, vendor onboarding, claims intake, or contract review. Then build the smallest eval harness that can answer whether the agent is ready.

The first version should include:

1. A golden set of 50 to 100 real cases. Include happy paths, edge cases, policy exceptions, and recent failures.
2. A behavior spec. Write down the required tool calls, forbidden actions, approval gates, data sources, and completion criteria.
3. Deterministic checks. Use code for anything that can be checked exactly: amounts, dates, IDs, status transitions, duplicate writes, missing citations, and forbidden calls.
4. Human or model review. Use rubrics for quality areas that need judgment: helpfulness, completeness, tone, ambiguity handling, and escalation quality.
5. Trace capture. Log the full path from input to action, with privacy controls and retention rules.
6. A promotion gate. Block changes that miss the threshold, and review failures before retrying.
7. A weekly case review. Add production failures and reviewer edits back into the dataset.

This is the part that turns a good demo into a deployable workflow.

Anthropic's guidance on building effective agents draws a useful distinction between workflows, where LLMs and tools follow predefined paths, and agents, where the model controls more of the process and tool use. That distinction should guide your eval depth. The more freedom the agent has, the more you need trajectory checks, tool-call assertions, replayable traces, and human sampling.

For many mid-market workflows, the right answer is not maximum autonomy. It is a bounded agent inside a measured workflow. Give it scoped tools. Define pass and fail. Let it ask for help. Measure every change.

The PE Angle: Evals Make AI an Operating Asset

Private-equity operators do not need another AI pilot that performs well in a board deck and then stalls in the business. They need automation that can be measured, improved, and handed over.

Production evals make that possible because they turn AI reliability into an owned asset: workflow cases, scoring rules, trace history, release gates, and the improvement loop. That matters during scaling, staff turnover, model changes, vendor negotiations, and exit diligence. A prompt is hard to diligence. A test suite, trace store, and pass-rate history are much easier to inspect.

This is where OpenNash fits. We build agents around the operating layer: workflow mapping, tool boundaries, trace capture, eval cases, release gates, dashboards, and the weekly review loop. The output is a measurable workflow that the client owns.

For support, that may mean 100 historical tickets converted into eval cases, with assertions for routing, policy, citations, refund gates, and CRM writes. For finance, it may mean invoice exceptions with checks for amounts, vendors, approval limits, and audit notes. For compliance, it may mean document folders with clause extraction, citations, risk tags, and reviewer rubrics.

The question is not "which model is best?" The better question is: which workflow can we measure well enough to trust, and what release bar must the agent clear before it touches production?

What to Ask Before Scaling

AI agents do not fail only by hallucinating. They fail by picking the wrong tool, skipping a step, using stale context, retrying badly, ignoring a policy, acting without approval, or succeeding in a way nobody can reconstruct.

Production evals exist because those failures are workflow failures. They require workflow tests.

Before expanding an agent beyond a pilot, ask for the evidence:

• Show the golden dataset.
• Show the trace for a passed case and a failed case.
• Show the release threshold.
• Show which failures block launch.
• Show how live failures become new evals.
• Show who owns the suite after deployment.

If a vendor or internal team cannot show that, they may still have a promising demo. They do not yet have a production workflow.

The release decision should be that plain: here is what the agent did, here is how we scored it, here is what failed, here is what changed, and here is why it is ready for more work.