AI Agents Part 2: Harness and CLI

AI Agent Engineering From Scratch

• AI Agents Part 1: Architecture
• AI Agents Part 2: Harness and CLI
• AI Agents Part 3: Tools and APIs
• AI Agents Part 4: MCP
• AI Agents Part 5: Context and Memory
• AI Agents Part 6: Agentic RAG
• AI Agents Part 7: Evals and Observability
• AI Agents Part 8: Security and Deployment

Part 1 explained the basic agent loop. A model chooses a next step, while application code controls the available actions.

That application code is the agent harness. It controls each run from the first request to the final result.

The harness matters because a model call cannot manage a complete task by itself.

Suppose a user asks an agent to investigate an invoice and prepare a refund. The model can interpret the request and suggest a next step.

The application must still answer practical questions:

• Which customer started the request?
• Which invoice may the agent read?
• Is the requested refund allowed?
• Did the payment service complete the write?
• How many times can the agent retry?
• What should the user see if approval is required?
• Which events must the application record?

The harness answers these questions with code and stored state. It turns model suggestions into controlled application behavior.

Why a model call is not enough

A model call has an input and an output. An agent run can have many steps, tools, failures, and pauses.

The model might request a tool with an invalid argument. A service might time out after it completes a payment. A policy might require human approval.

A chat transcript cannot resolve these cases. The application needs rules, state, and evidence that exist outside the model.

A useful harness provides five groups of controls:

1. Execution. It starts the run, calls the model, validates responses, and calls approved tools.
2. State. It stores progress, results, pending actions, and errors in defined fields.
3. Limits. It restricts time, model calls, tool calls, tokens, output size, and cost.
4. Recovery. It classifies failures, controls retries, and resumes paused runs.
5. Inspection. It records events so developers and operators can explain each result.

A framework can provide code for some of these controls. Your application must still define their meaning for your product.

The harness controls each step

The model requests an action. The harness checks the request before the application executes it.

A basic run follows this sequence:

1. Create a run ID and store the authenticated user.
2. Select the instructions, state, tools, and evidence for the current step.
3. Send that information to the model.
4. Convert the model response into a defined data structure.
5. Reject missing fields, invalid values, and unsupported actions.
6. Check whether the user can perform the requested action.
7. Call the approved tool and record its result.
8. Update the run state.
9. Continue, pause, or stop.

This sequence creates a clear authority boundary. The model can request an action. The harness controls whether the action occurs.

Build the loop as ordinary application code

The main loop does not need a complex framework. Its responsibilities should remain visible even when a framework implements some of them.

This TypeScript-style example shows the control flow:

async function runAgent(runId: string, signal: AbortSignal) {
  let state = await runs.load(runId);

  while (state.status === "running") {
    assertWithinBudget(state);
    const context = await buildContext(state);
    const raw = await model.decide(context, { signal });
    const decision = DecisionSchema.parse(raw);

    await events.append(runId, {
      type: "model_decision", decision,
      promptVersion: context.promptVersion
    });

    const outcome = await dispatch(state, decision, signal);
    state = transition(state, decision, outcome);
    await runs.save(state);
  }

  return state;
}

Each function has one job. buildContext selects model input. The schema parser checks the response shape.

dispatch checks policy and calls an allowed capability. transition applies a defined state change. The repository stores the new state.

Keep the loop small. Put business rules, API adapters, and context selection in separate modules with their own tests.

The code also needs a final error boundary. It should convert known failures into run statuses and preserve unexpected failures for investigation.

Store state as data

The harness needs structured state because a conversation transcript does not contain reliable application facts.

A small run can contain these fields:

run_id
status
current_step
request
authenticated_user_id
verified_resource_ids
model_call_count
tool_call_count
deadline
pending_action
final_result
last_error

Each task needs different fields. Store every value that the application needs to continue, authorize, inspect, or evaluate a run.

Read business data from its authoritative source. For example, read payment status from the payment service.

Store the agent's progress in the run state. Send the model only the fields that it needs for the current step.

This separation matters after a crash. The application can reload the stored state and continue without asking the model to reconstruct events.

Add a version number to the state. Use an atomic compare-and-save operation when more than one worker can process runs.

save(run_id, expected_version=7, next_version=8)

The save must fail if another worker already wrote version 8. This check prevents two workers from advancing the same run.

Use a lease when a worker needs temporary ownership. Give the lease an expiry so another worker can recover an abandoned run.

Version checks protect internal state. Idempotency keys protect external writes. A production harness usually needs both.

Give every run a clear status

A run status tells the application what happened and what it can do next.

Useful statuses include:

Each status controls the next application action. It can change the user message, API response, and available resume options.

The service that stores a record is its system of record. Only that service can confirm that a write succeeded.

The model can explain why information is missing. Application code must decide whether the run enters needs_input.

Allow only defined state changes

A state transition moves a run from one status to another. Define the allowed transitions in code.

running -> completed
running -> needs_input
running -> needs_approval
running -> denied
running -> budget_exhausted
running -> failed

needs_input -> running       after new input
needs_input -> cancelled     after authorized cancellation
needs_approval -> running    after a valid approval
needs_approval -> denied     after rejection
needs_approval -> expired    after proposal expiry
running -> cancelled         after authorized cancellation

Reject every transition that is absent from this list. Add a transition only after you define and test its behavior.

These checks prevent two common failures. A denied run cannot restart because the model rephrased its request.

A terminal-state check rejects a retry against the same stored run. Service-level idempotency still protects the external write after races or crashes.

Test the transition code without a model. State changes are normal application logic.

Walk through a complete run

Continue the duplicate-charge example from Part 1. The CLI starts a run with this command:

agent run refund-request.json --format json

The harness writes an event before it calls the model. It then advances the run through explicit states.

The process can stop after step 5. The stored run has a pending proposal and does not depend on a worker staying alive.

Later, an approval command resumes it:

agent approve proposal_123 --run run_abc123 --json

The harness verifies the approver, proposal hash, expiry, and current run version. It then changes needs_approval to running.

Before the commit, it checks the run budget again. The payment adapter sends the proposal ID as the idempotency key.

The run reaches completed only after the payment service returns a durable receipt. A model message that says "refund complete" cannot set that status.

Enforce limits in code

A prompt can ask a model to use fewer tools. The prompt cannot guarantee that behavior.

A run budget is a limit that application code enforces. Set limits for:

• model calls
• tool calls
• elapsed time
• input and output tokens
• generated code execution time
• retrieved documents
• model cost
• output size

Different tasks need different limits. A classification task might allow one model call and no tools.

A research task might allow several searches but no write tools. A deployment task might allow more time and require checkpoints.

Check the deadline before each network request. Also give the model client and each tool a cancellation signal.

The harness can then stop a request that has already started.

When a run reaches a limit, set its status to budget_exhausted. Do not request an unsupported final answer from the model.

Retry only safe operations

Do not retry every error. A retry can repeat an external action that already succeeded.

Classify failures before you choose a response:

• invalid model output
• model refusal
• timeout
• rate limit
• authentication failure
• permission denial
• missing resource
• temporary service failure
• uncertain write result

A timed-out read may be safe to retry once. Invalid tool input should return to validation or request new input.

A permission denial should stop the run. A timed-out write needs an additional check.

Give each write an idempotency key. This unique value lets the target service recognize a repeated request.

Before a retry, ask the service whether it already processed that key. Return the stored result when the first request succeeded.

Set a maximum retry count. Record every retry and its result.

Retry only when the target API documents that the operation is safe to repeat.

Recovery also needs a rule for errors with unknown outcomes. A network timeout during a write does not tell you whether the service applied it.

Move the run to a recovery state or keep it paused. Query the source service with the idempotency key before another write attempt.

Never ask the model to guess whether the action completed.

Record a trace

A trace is an ordered record of a run. It shows the available input, requested actions, results, state changes, and final status.

Record events such as:

• run started
• model requested
• model returned
• tool requested
• authorization decided
• tool returned
• state changed
• approval requested
• budget reached
• run finished

Record the version of every component that can change a result. These components can include the model, prompt, tool definitions, retrieval index, and policy.

Also record latency and resource use. Remove secrets and sensitive fields before storage.

Do not store hidden model reasoning. Store the model response, tool requests, tool results, and state changes that the application can verify.

A trace helps you inspect a run. Deterministic reconstruction applies stored events to rebuild state without calling the model or tools.

A test replay feeds recorded model and tool outputs through current application code. It checks transitions without repeating external work.

Re-executing live tools is a new run, not a replay. It can observe different data or repeat side effects.

A useful event envelope can look like this:

{
  "event_id": "evt_009",
  "run_id": "run_abc123",
  "sequence": 9,
  "type": "tool_completed",
  "recorded_at": "2026-08-24T16:31:02Z",
  "actor": "tool:refund_commit",
  "input_ref": "proposal_123",
  "output_ref": "receipt_789",
  "state_version": 8
}

Store large or sensitive payloads separately and reference them by ID. Apply access control and retention rules to both the event and its payload.

Events need a monotonically increasing sequence within one run. Timestamps alone cannot reliably order events from several workers.

Add a command-line interface

A command-line interface, or CLI, gives people and software a direct way to use the harness.

A useful CLI can support these commands:

agent run "Summarize the open issues for account 123" --json
agent inspect run_abc123
agent approve proposal_xyz
agent evaluate evals/golden-cases.jsonl

Write result data to standard output (stdout). Write warnings and error details to standard error (stderr).

This separation lets scripts read results without parsing diagnostic messages. Return meaningful exit codes, and make --help work without credentials.

Make the CLI, web interface, and chat interface call the same application service. This design keeps run behavior consistent across all interfaces.

The CLI also supports a short development cycle:

1. Run one case.
2. Inspect the state and trace.
3. Change the code.
4. Run the case again.
5. Add the case to the test suite when it represents a useful failure.

Continuous integration jobs, scheduled tasks, shell scripts, and other software can also call the CLI.

Treat the CLI output as a public contract. A human-readable table is useful at a terminal, but automation needs stable JSON.

{
  "run_id": "run_abc123",
  "status": "needs_approval",
  "proposal_id": "proposal_123",
  "next_actions": ["approve", "deny"]
}

Use exit code 0 for a successfully handled command. Use distinct nonzero codes for invalid input, denied access, and unavailable services.

Do not place secrets, raw prompts, or private tool results in default CLI output. Make detailed inspection a separate authorized command.

The CLI should also support a fake model and fake tools. This makes a complete run deterministic in tests.

agent run fixtures/refund.json --model fake --tools fake

That command can verify every event, transition, and exit code without network access.

Common harness failure modes

Test these failures directly. Do not wait for a live model to produce them by accident.

Build the first harness in this order

Start with the smallest harness that can expose a complete run:

1. Define the run state and statuses.
2. Create a test model that returns a known response.
3. Add one model call with a time limit.
4. Validate the model response.
5. Record trace events.
6. Expose the run through the CLI.
7. Add one read-only tool.
8. Add limits for tools and time.
9. Add approval and recovery when the task requires them.

After each step, add one successful test and one failure test. Both tests should run without a live model.

For every status change, record the previous status, the new status, and the event that caused the change.

Harness review checklist

• Does every run have a durable ID, status, and version?
• Can two workers advance the same run?
• Does every loop iteration check budgets before network work?
• Are model responses schema-validated before dispatch?
• Can the model change an authenticated identity or approval record?
• Are transitions defined and tested without a model?
• Can a paused run resume after the process exits?
• Are write retries protected by source-service idempotency?
• Does completion require a verified result or receipt?
• Can an operator inspect events in their exact order?
• Does cancellation reach the model client and every tool?
• Can the CLI run a complete offline test case?

The harness is ready for more tools when these controls work with one tool and one failure path.

Previous: AI Agents Part 1: Architecture

Next: AI Agents Part 3: Tools and APIs explains how the harness connects an agent to external systems.