AI Agents Part 7: Evals and Observability

AI Agent Engineering From Scratch

• AI Agents Part 1: Architecture
• AI Agents Part 2: Harness and CLI
• AI Agents Part 3: Tools and APIs
• AI Agents Part 4: MCP
• AI Agents Part 5: Context and Memory
• AI Agents Part 6: Agentic RAG
• AI Agents Part 7: Evals and Observability
• AI Agents Part 8: Security and Deployment

An agent can return the right answer after following an unsafe or unreliable path. It might read restricted evidence, skip approval, or report a write that never occurred. A test that checks only the final answer will miss these failures.

An eval is a repeatable test of an agent. It gives the agent a defined task and checks its behavior against stated criteria. Agent evals examine both the result and the steps that produced it.

Observability is the ability to inspect a run after it happens. The system records a trace for each run. A trace contains the input, model decisions, retrieved evidence, tool calls, state changes, and final output.

The trajectory is the ordered path recorded in that trace. A grader checks one condition in the trajectory or final outcome. Together, evals and traces show whether an agent succeeded and why.

Follow one agent through an eval

Consider a support agent that handles refund requests. It can read an order, retrieve the refund policy, propose a refund, request approval, and submit the refund.

A user asks:

Refund order 4812. The delivery arrived nine days late, and the customer paid £68.

A successful final message could say that the refund was approved and submitted. That answer alone does not prove the run was correct. The agent also had to:

1. read order 4812 from the correct customer account.
2. retrieve the policy version active on the delivery date.
3. calculate an amount allowed by that policy.
4. request approval because the amount exceeds the automatic limit.
5. submit one refund after approval.
6. report the receipt returned by the payment service.

The test therefore needs three kinds of evidence. It needs the final answer, the trajectory, and the resulting state of the payment system.

A compact trace might look like this:

{
  "run_id": "run_0194",
  "task_id": "late_delivery_refund",
  "release": "agent-2026-08-24.3",
  "principal": "support_user_17",
  "events": [
    {"type": "tool_call", "tool": "orders.get", "args": {"order_id": "4812"}},
    {"type": "tool_result", "status": "ok", "customer_id": "cust_77", "total": 68},
    {"type": "retrieval", "source_id": "policy_2026_04", "passage_ids": ["p18"]},
    {"type": "proposal", "action": "refund", "amount": 68},
    {"type": "approval", "approval_id": "apr_552", "amount": 68},
    {"type": "tool_call", "tool": "refunds.create", "idempotency_key": "refund:4812:1"},
    {"type": "tool_result", "status": "ok", "receipt_id": "rf_901"}
  ],
  "final_status": "completed"
}

This trace gives graders something concrete to inspect. One grader can verify the policy source. Another can confirm approval. A third can query the test payment service for receipt rf_901.

The same final answer could hide several failures. The agent might use an old policy, read another customer’s order, or submit the refund twice. Each failure needs a separate check.

Decide what the eval must protect

Public benchmarks measure performance across a broad set of tasks. Product evals test the work that users depend on.

Start with failures that could block a release, expose private data, create an incorrect record, or damage user trust.

The HowToEval guide describes this approach as raising the floor.

This approach focuses on error analysis. Find failures that damage trust, locate their first cause, and make them less likely.

Start with a small set of tasks that represent important user work:

• the common path users depend on.
• the expensive mistake.
• the permission boundary.
• the case that must ask for approval.
• the situation where the agent should say it does not know.
• the failure that would stop you from shipping.

An average pass rate can hide repeated failures in one task group. Report each important group as well as the overall rate.

Define the units

Clear terms prevent confusing reports.

• A task is one test case. It includes an input, test data, an authenticated identity, and an expected outcome.
• A trial is one run of a task.
• A trace is the stored record of one trial.
• A trajectory is the ordered sequence of decisions, retrievals, tool calls, results, and state changes within that trial.
• An outcome is the final answer and the resulting environment state.
• A grader checks one stated condition in the trajectory or outcome.
• An eval suite is a versioned collection of tasks and graders.

One task may need several trials because the model can choose different paths.

Run each trial in an isolated test environment. Reset its files, databases, clock, queues, and test data before the next trial.

Otherwise, one trial can change the next trial.

Start with golden cases

A golden case is a small test for behavior that the agent must preserve.

Start with five to ten cases that cover important user workflows. Each case needs a clear contract:

task input
authenticated identity
test data
expected final status
required evidence
allowed actions
forbidden actions
expected external changes
maximum model calls
maximum tool calls

Store the contract as data rather than prose in a test document. For example:

id: late_delivery_refund_requires_approval
input: "Refund order 4812 because it arrived nine days late"
principal: support_user_17
fixture: refund_fixture_v4
expected:
  final_status: completed
  required_sources:
    - policy_2026_04
  required_tools:
    - orders.get
    - refunds.create
  forbidden_tools:
    - orders.change_owner
  max_refund: 68
  approval:
    required: true
    amount: 68
  external_state:
    refund_count: 1
    refund_total: 68
budgets:
  model_calls: 8
  tool_calls: 10
  elapsed_ms: 30000

A fixture is the controlled environment for the task. It includes the order, customer, policy, approval response, clock, and payment-service behavior.

Version the fixture with the case. A policy change should not silently alter the expected result of an older release test.

Run each case through the same application code that production uses. If an important case fails, find the cause before release.

Golden cases should remain small enough that engineers read the failures. A suite no one inspects becomes a noisy ritual.

Add a case when it protects a meaningful behavior or a recurring failure class. Do not keep every one-off production oddity forever.

Read trajectories before writing graders

A failed answer tells you that something went wrong. The trajectory tells you where.

Suppose an agent gives an outdated policy answer. The first failure might be:

1. ingestion kept the old version.
2. retrieval missed the current passage.
3. permissions removed the only useful source.
4. the context builder omitted required evidence.
5. the model ignored correct evidence.
6. the answer used the evidence but attached the wrong citation.

One "answer incorrect" grader cannot distinguish these causes.

Read a sample of complete trajectories and label the first meaningful failure. Build a small taxonomy from what you observe:

wrong_tool
invalid_arguments
permission_denial_ignored
required_evidence_missing
denied_evidence_retrieved
unsupported_claim
citation_mismatch
side_effect_without_approval
duplicate_side_effect
budget_exceeded
handoff_missing

Keep labels narrow enough that two reviewers can apply them consistently. Use each failure category to identify the component that needs a fix.

Prefer deterministic graders

If code can verify a condition, use code.

A deterministic grader uses code to check a condition. It produces the same result when it receives the same data.

Deterministic graders can check:

• output schema.
• final state.
• database records.
• file changes.
• tool names and normalized arguments.
• permission decisions.
• proposal amount.
• approval presence.
• idempotency receipt.
• source presence, allowed source IDs, source versions, and labeled claim mappings.
• token, time, call, and cost budgets.
• environment reset.

These graders are repeatable and easy to debug.

Open-ended citation support needs semantic judgment. Use a calibrated model grader or human reviewer to decide whether a source supports an arbitrary claim.

A grader should return more than a Boolean value. Include the evidence that explains its decision:

{
  "grader": "approval_matches_refund",
  "version": "2",
  "result": "PASS",
  "evidence": {
    "approval_id": "apr_552",
    "approved_amount": 68,
    "submitted_amount": 68,
    "consumed_once": true
  }
}

The grader can implement four direct checks:

find the refunds.create event
find the approval consumed before that event
compare order, currency, and amount
confirm one payment receipt exists for the idempotency key

Return ERROR when the grader cannot read the trace or test service. Do not treat a broken grader as a failed agent. The release system must distinguish both conditions.

Use three result states when appropriate:

Keep the grader independent from the code it checks. If the tool and grader use the same faulty refund calculation, they can agree on the wrong amount.

Keep important criteria separate. Do not blend safety, correctness, and style into one weighted score. An agent that answers correctly after reading denied data still fails.

A task may allow several safe trajectories. Grade exact call order only when policy or the product requires it. Otherwise grade the outcome and invariants.

Use model graders for subjective criteria

Some criteria are difficult to express as code:

• Is the explanation clear?
• Does the response acknowledge uncertainty?
• Is a refusal helpful without revealing private details?
• Does the answer address the user's actual concern?
• Is the summary faithful to a long source?

A model grader uses another language model to assess a response. Use it only when code cannot express the criterion reliably.

Use one criterion at a time. Ask for a small label set such as PASS, FAIL, or UNKNOWN. Give the model grader only the evidence it needs.

Before using a model grader as a release gate:

1. collect representative human labels.
2. compare judge and human decisions.
3. inspect false passes and false failures.
4. test high-risk task groups separately.
5. version the grader model and prompt.
6. define what happens on UNKNOWN.

Test the model grader against representative human decisions before it controls a release.

Test each layer before the whole agent

End-to-end tests matter, but they can be slow and hard to diagnose. A useful eval system has several layers.

Run cheap component tests on every change. Run the smaller end-to-end suite before merge. Run expensive repeated trials before a production release when the risk requires them.

This layering reduces diagnosis time. If retrieval fails before the model runs, changing the prompt will not repair the missing passage.

It also prevents one broad score from hiding the broken component. Keep a release gate for each invariant that can cause unacceptable harm.

Design model graders as measured instruments

A model grader introduces variation and bias into the eval system. Treat it as a measured instrument, not an authority.

Start with a narrow rubric. For a support response, the rubric might check whether the explanation matches the evidence and states uncertainty when evidence conflicts.

Give the grader:

• the user request.
• the final response.
• only the source passages needed for the decision.
• one criterion and its definitions.
• labeled examples near the decision boundary.
• an explicit UNKNOWN option.

Do not show the grader an expected label. Do not ask one prompt to score correctness, tone, safety, and completeness at once.

Measure agreement on a held-out set of human-labeled examples. Inspect disagreements by category. A high average agreement can still hide poor decisions on refusals or sensitive cases.

Version the grader prompt, model, examples, and inference settings. When any part changes, rerun the calibration set before comparing new agent results with old results.

Run repeated trials for the right reason

One successful trial shows that the agent can pass once. It does not establish reliability.

Repeat important tasks when model variation can change the trajectory. Preserve pairing when comparing two versions: run both variants on the same tasks, fixtures, and trial seeds where the provider supports them.

Report:

• number of tasks.
• number of trials per task.
• success by slice.
• distribution of latency and cost.
• important failure counts.
• uncertainty around aggregate rates.

In both metric names, k is the number of trials.

pass@k measures whether at least one of the k trials succeeds. It tests whether the agent can solve a task after several attempts.

pass^k measures whether all k trials succeed. It tests whether the agent performs consistently.

A workflow can have a high pass@k result and a low pass^k result. Production workflows often require consistent success.

There is no universal trial count. Choose it according to the release decision, the failure rate you care about, risk, runtime, and cost.

Suppose the refund task runs five times. Three trials pass, one skips approval, and one exceeds the tool-call budget.

The per-trial success rate is 60 percent. pass@5 is true because at least one trial passed. pass^5 is false because all five did not pass.

Neither metric should replace the failure counts. The approval violation remains a release blocker even if many other tasks pass.

When comparing release A with release B, keep the task set and fixture versions fixed. Run both versions enough times to expose meaningful variation.

Compare paired outcomes for each task. An aggregate improvement can hide a regression in the highest-risk group.

Also compare distributions rather than averages alone. A mean latency of four seconds can hide a small group of runs that take one minute.

Separate offline evals from production observation

Offline evals run in a controlled environment. They are best for:

• golden cases.
• regression testing.
• permission and safety invariants.
• model or prompt comparisons.
• deterministic side-effect checks.
• release gates.

Production traces show how the agent behaves with real requests, data, tools, and failures. Use them to find:

• new request patterns.
• tool and integration failures.
• context gaps.
• latency and cost distributions.
• user corrections.
• handoffs and abandonment.
• rare failure clusters.

Before launch, you are making an informed guess about failures. After launch, users provide the cases you did not imagine.

Build traces for diagnosis

A trace should reconstruct the run without exposing hidden reasoning or unnecessary private data. Record observable decisions and their inputs instead.

Use a stable event envelope:

{
  "event_id": "evt_044",
  "run_id": "run_0194",
  "parent_event_id": "evt_043",
  "time": "2026-08-24T10:14:22Z",
  "type": "tool_result",
  "component": "refunds.create",
  "release": "agent-2026-08-24.3",
  "duration_ms": 381,
  "status": "ok",
  "data_ref": "trace-data://run_0194/evt_044",
  "policy": {"decision_id": "pd_802", "version": "refund-policy-12"}
}

The event identifies the run, its parent, component, version, duration, status, and policy decision. Put large or sensitive payloads in protected storage with separate access controls.

Record normalized tool arguments after validation. Record a hash or reference when raw content should not enter the main trace store.

Propagate the run ID across model, tool, queue, and service boundaries. Without that identifier, investigators must join events by timestamps and guesses.

Choose retention by data class. A debugging trace can contain order details, source passages, or user text. Limit access, redact before storage, and delete data on a defined schedule.

Do not log access tokens, session cookies, complete secrets, or hidden credentials. A trace system with broad access can become a second data leak.

Scale review with volume

At low volume, read most runs. This is how the team develops a feel for the system.

As volume grows:

1. sample complete trajectories.
2. filter obvious technical failures.
3. cluster similar errors.
4. track recurring issues.
5. monitor high-risk signals.
6. compare changes through controlled experiments.

Use automation to select runs for review. Engineers should continue to inspect complete traces and raw examples.

Keep a path from every dashboard number back to the underlying run. Aggregates without examples are difficult to act on.

Use the failure loop

A useful production loop is short:

1. Find a meaningful failure.
2. Reproduce it locally.
3. Identify the first broken component.
4. Add or update a focused regression case.
5. Fix the component.
6. Run the relevant suite repeatedly.
7. Release behind an appropriate gate.
8. Watch the affected production slice.

Fix the failure class across several forms of the request.

Prune evals that no longer protect meaningful behavior. Twenty cases that regularly inform release decisions are better than two hundred ignored checks.

Record the data needed to explain a run

Collect enough data to answer:

• Which model and configuration ran?
• Which instructions and tool definitions were active?
• Which principal and policy version applied?
• Which sources entered context?
• Which tools were requested and executed?
• Which state changes occurred?
• Where did time and tokens go?
• What stopped the run?
• Did the authoritative database or service confirm the outcome?

Do not collect sensitive content without retention and access rules.

Remove sensitive data before storing the trace. Do not rely on a model instruction to remove secrets.

The eval system should consume these traces rather than inventing a second representation of the run.

Common eval failures

Eval systems can create false confidence. Watch for these failure modes:

Build the eval system in this order

Use this sequence for a first production eval system:

1. Choose five to ten tasks tied to user value or serious risk.
2. Create isolated, versioned fixtures for each task.
3. Define the expected final state and forbidden actions.
4. Add stable run IDs and structured trace events.
5. Read complete trajectories and create a failure taxonomy.
6. Write deterministic graders for objective invariants.
7. Add a calibrated model grader only for subjective criteria.
8. Repeat the highest-risk tasks and report failures by group.
9. Turn critical criteria into release gates.
10. Sample production traces and feed meaningful failures back into the suite.

For the refund agent, the first release gate should be blunt. No trial may cross an account boundary, execute without required approval, or create more than one refund.

The next article turns those invariants into runtime controls. It covers identity, authorization, approvals, isolated execution, recovery, and staged deployment.

For broader OpenNash coverage, see How to Eval AI Agents in 2026 and Production Evals for Agentic Systems.

Previous: AI Agents Part 6: Agentic RAG

Next: AI Agents Part 8: Security and Deployment turns the failure model into permissions, approvals, isolation, recovery, and release controls.