Your Agent Failed in Prod. Can You Replay It?

An AI agent does not fail like a normal web form.

A normal form rejects the wrong input, throws an exception, or writes a bad value that you can often reproduce with the same request. An agent can fail with a clean 200 response, a plausible final answer, and a database write that looks valid to every dashboard. The customer notices first because the outcome is wrong, not because the system crashed.

A common failure pattern makes the risk obvious: a user asks an agent to sell about $1,000 worth of a stock. The agent sends the raw number 1000 into the quantity field. At $190 per share, a small trade request turns into a $190,000 mistake. The broker API returns 200 OK. Latency is fine. Error rate is zero. The business outcome is a mess.

For mid-market companies and private-equity operators, this is the line between an AI pilot and an operating asset. If the agent touches support, billing, inventory, contracts, finance, claims, onboarding, or CRM records, reliability cannot mean "the model usually gives a good answer." Reliability means the team can answer four questions after every run:

• What did the agent see?
• What did it decide?
• What did it do?
• How do we replay that path without asking the model to recreate it?

If you cannot answer those questions, you do not have a production agent. You have a fast employee with no audit trail.

The Wrong Debugging Reflex

The first instinct after an agent incident is to grab the prompt from logs, rerun it locally, and see if it breaks. That feels like normal debugging. It also fails at the exact moment you need it most.

The rerun may work. Then it may work again. Then the team decides the original failure was a fluke and ships a prompt tweak. What actually happened is worse: the only run that mattered is gone.

The reason is simple. The prompt was never the whole input. A production agent run is a bundle of moving pieces: model version, system prompt, user message, retrieval chunks, memory, tool schemas, tool responses, account permissions, hidden state, environment variables, retry behavior, timestamps, and records in external systems. A support agent may retrieve a policy document. A finance agent may inspect invoices, purchase orders, and vendor history. A sales agent may call CRM and email tools. A healthcare intake agent may branch on patient eligibility. The prompt is one line in a much longer transaction.

The second reflex is to turn temperature to zero and hope determinism arrives. It does not. Greedy decoding reduces one source of variation, but it does not make the full system reproducible. Recent research on LLM inference, including work on determinism under dynamic batching and floating point behavior, shows why identical inputs can still behave differently when inference systems optimize for throughput. Even if the model output were perfectly repeatable, the agent still depends on retrieval state, tool behavior, business data, permissions, and code version.

The practical lesson is not "give up on reproducibility." It is "stop trying to reproduce a failure by wishing the model would do the same thing twice." Record the run while it happens. Then replay the recorded state transition.

Trace the Boundary, Not Just the Prompt

The useful unit of observability is the boundary. A boundary is any step where the agent receives input and emits output: a planner call, model call, retrieval call, tool invocation, guardrail, approval step, write operation, or final response.

This matches how modern tracing works outside AI. OpenTelemetry describes traces as records of a request path through a system, broken into spans that show the work performed along the way. Agent systems need the same shape, with additional fields for model, prompt, context, tool, and policy state.

The difference is that agent traces need business semantics, not packet-level noise. You do not need every TCP detail. You need to know that the planning node interpreted "$1,000 worth" as quantity: 1000, that the order tool accepted the request, that no approval gate ran, and that the final response told the user the sale was complete.

Good traces capture:

Tools such as OpenAI Agents SDK tracing and LangSmith tracing reflect the same shift. Once agents become multi-step systems, debugging requires a trace of the run, not a screenshot of the final answer.

For business systems, traces also need retention, redaction, and access control. A trace can contain customer data, pricing data, financial records, contracts, and employee notes. The audit layer should be useful to engineering, operations, compliance, and customer support without turning every trace store into an ungoverned copy of sensitive data.

Replayability Is Different From Determinism

Bitwise determinism asks for the same input to return the same output. Replayability asks for enough of a past run to debug, test, and repair it.

That distinction matters. You do not need the model to produce the exact same token stream tomorrow. You need the system to preserve what happened yesterday. If the planning model created the bad order request, record that request and replay it into the downstream code. If the retrieval layer returned stale policy, record the retrieved chunks and document versions. If the approval gate failed to run after a retry, record the state transition that skipped it.

Replay turns an agent incident into a controlled engineering artifact:

1. Load the trace from the failed run.
2. Stub the probabilistic steps with the recorded outputs.
3. Run the changed deterministic code live.
4. Assert that the bad action is blocked, routed, corrected, or reversed.
5. Keep the trace as a regression test.

This is the bridge between observability and testing. It is also where agent operations start to look like real software operations. The OpenNash post on turning production failures into eval cases makes the same point from the incident-review side: a failure that does not become a test will probably come back.

Replayable traces make that rule enforceable. The trace is no longer just a debugging aid. It becomes the fixture for a test case. The fix is not "we added a sentence to the prompt." The fix is "the trace that sold 1,000 shares now fails before the tool executes, and CI will catch any change that reopens the path."

The Test Suite Should Mix Hard Checks and Behavioral Evals

Agent testing needs two layers.

The first layer is deterministic testing. These are the parts of the workflow that should behave like ordinary software: tool schema validation, permissions, thresholds, approvals, idempotency keys, write constraints, recovery actions, and policy enforcement. A stock order tool should not accept a vague dollar amount as share quantity. A refund tool should not exceed a limit without approval. A CRM tool should not update two accounts when the match confidence is low.

These checks should run fast and fail loudly. They do not need a model judge. They need assertions.

The second layer is behavioral evaluation. This covers the parts that are harder to reduce to one boolean: tone, helpfulness, prioritization, route selection, reasoning quality, citation quality, and whether the agent asked a clarifying question when the request was ambiguous. These evals can use human review, model-graded checks, or sampled production review.

The pattern matters more than the tooling. A failed trace should tell you which layer broke. If the model selected the wrong tool, create an eval for routing. If the tool accepted unsafe input, add deterministic validation. If retrieval returned stale context, test the index and freshness path. If a human approval was skipped, test the orchestration state machine.

This is where Harbor-style agent eval infrastructure fits well. A repeatable task, environment, and verifier let a team test whole workflows, while replayable production traces let the suite grow from real incidents. One is the lab bench. The other is field evidence.

Audit Logs Are a Business Control

Agent audit logs are often treated as an engineering detail. That is a mistake. They are a business control.

NIST's AI Risk Management Framework organizes AI risk work around governance, mapping, measurement, and management. For an operating company, those words become practical questions: who owns this workflow, what harm can it cause, how is performance measured, and what happens when it fails?

Audit logs answer part of each question. They show whether the agent stayed within its scope, whether approvals ran, whether policy checks were bypassed, whether a tool wrote to the right record, and whether recovery work happened. Without that evidence, the company is left with anecdotes.

Security guidance points in the same direction. The OWASP Top 10 for LLM Applications includes risks such as prompt injection, sensitive information disclosure, improper output handling, excessive agency, vector and embedding weaknesses, misinformation, and unbounded consumption. Every one of those risks is harder to manage if you cannot inspect what the agent did step by step.

For PE-backed and mid-market operators, the audit question is not theoretical. The first AI agent most companies deploy often sits near a system of record: ticketing, ERP, CRM, billing, or document management. That means the agent's actions can change revenue, cash, customer trust, or compliance posture. If the post-incident answer is "we cannot tell why it did that," the value case for automation weakens fast.

Recovery Paths Must Be Designed Before Launch

The stock-trade example is dramatic because the tool action is irreversible in spirit, even if there are financial processes for correcting it later. Many business agent actions are similar. They may not be market orders, but they still create real cleanup work:

• A support agent promises a refund the company should not honor.
• A billing agent applies the wrong credit memo.
• A procurement agent approves the wrong vendor change.
• A sales agent updates the wrong opportunity stage.
• A compliance agent marks a contract review complete with a missing clause.
• An HR agent sends a policy answer to the wrong employee population.

Recovery cannot be improvised after the run. It belongs in the workflow design.

Every production agent should have an action register with four fields: reversibility, approval rule, compensation path, and owner. A reversible action may be undone directly. A compensating action may require a new transaction, customer notice, or finance adjustment. A non-reversible action may require pre-approval every time. The owner is the person or team paged when recovery is needed.

The trace should support recovery, not just diagnosis. If an agent writes to a record, capture the before and after state. If it calls an external API, capture the idempotency key and response. If it notifies a customer, capture the exact message and channel. If a human approves, capture the decision and reason. Google's SRE guidance on postmortem culture is useful here because the goal is not blame. It is learning enough from failure to make the system safer the next time.

The recovery path is also where humans belong. Human-in-the-loop review should not be a decorative "approve" button bolted onto the end. It should be a policy decision tied to value, ambiguity, reversibility, and customer impact. A $25 credit may be safe to automate. A $25,000 credit may need review. An ambiguous instruction should pause. A tool call that changes a regulated record should produce an audit event before and after the action.

The Operator's Minimum Standard

Here is the minimum standard before an agent touches production systems of record:

This is not bureaucracy. It is the operating layer that makes agent automation investable. A company that saves headcount with an agent but cannot explain failures has created hidden operational debt. A company that records, replays, tests, and recovers is building institutional memory.

The weekly review matters too. Every incident should be tagged by failure type: prompt, tool, retrieval, permission, routing, workflow, policy, or recovery. Over time, the pattern tells management where to invest. If tool failures dominate, improve schemas and validation. If retrieval failures dominate, fix document freshness and citations. If approval failures dominate, fix orchestration. If policy failures dominate, move rules out of prose and into code.

This is the practical version of evaluation-driven development. The eval suite is not a side project. It is the memory of every way the agent has been wrong.

Where OpenNash Fits

OpenNash builds production agents for operators who care about the business process, not the demo. We start with the systems of record: which tools the agent can call, which fields it can write, which actions need approval, and which past incidents should become the first eval cases.

Then we instrument the boundaries. Model calls, retrieval calls, tool calls, policy checks, approvals, and writes all need structured records. We keep the traces useful enough for debugging and controlled enough for sensitive data. We connect incident review to the test suite so the agent improves from real failures rather than synthetic examples alone.

The goal is not "perfect AI." It is an operating layer for agent reliability:

• Audit logs that show what happened.
• Replayable traces that let engineering reproduce the state transition.
• Test cases that prevent old failures from returning.
• Approval gates that match business risk.
• Recovery paths that make mistakes containable.
• Dashboards that show reliability by workflow, not only token counts.

If an agent can write to CRM, billing, ERP, ticketing, finance, or customer systems, this is the release bar. Ship it only when failures are observable, replayable, testable, and recoverable. The next version should ship with a trace.