AI Agent Engineering From Scratch

An AI agent is a software application that uses a model to select its next action.

The application still receives requests, stores state, calls services, checks permissions, records events, and returns results.

The model is one component of the application. Ordinary code controls the model and the actions that it can request.

Begin with the job

Replace "Build an agent" with a task specification.

"Read a request, find the relevant account, explain the policy, and prepare an action for approval" is a useful specification.

It identifies the input, the required result, the records that the system can read, and the action that needs approval.

Before choosing a model or framework, write down:

  • what starts the task.
  • what a successful result looks like.
  • what the system may read.
  • what the system may change.
  • when a person must take over.
  • how the run stops.
  • what evidence you need afterward.

A broad task leaves important behavior undefined. A demo can still succeed when every service responds and the request is clear.

Test the difficult cases. A service can fail, a request can be unclear, or an action can change important data.

Start with one narrow task so that you can inspect every decision and failure.

Choose between a model call, workflow, and agent

These three designs solve different problems.

A single model call maps an input to an output. Classification, extraction, rewriting, and structured analysis often fit here. The application calls the model once and continues in code.

A workflow follows a path chosen by the developer. It may call models at several points, but code decides the sequence. Workflows are a good fit when the steps are known and the main challenge is handling data at each step.

An agent lets the model choose at least part of the sequence. It can decide which allowed tool to use, inspect the result, and choose another step.

Choose the simplest design that can complete the task. Use one model call when one response is enough.

Use a workflow when code can choose every step. Use an agent only when the model must choose a step during the run.

Each model-selected step creates more possible action sequences. You must test, secure, and explain each sequence.

The distinction becomes clearer when you draw the possible paths.

single call: request -> model -> result

workflow:    request -> extract -> look up -> calculate -> result
                         code chooses every arrow

agent:       request -> model -> tool -> model -> tool -> result
                         model chooses some arrows

The same application can use all three designs. A support system might use one model call to classify a request.

It can then run a fixed identity check. Only an unusual case might enter an agent loop to gather more evidence.

This mixed design is often easier to control than turning the entire process into one agent.

The smallest useful agent loop

Most agents can be reduced to five operations:

  1. Select the state, instructions, and tool results that the model needs for its next decision.
  2. Ask the model to choose an allowed action or finish.
  3. Validate the response.
  4. Execute the action and record the result.
  5. Stop or begin another turn.

In pseudocode:

state = start(request)

while state is active:
    decision = model(context_for(state))
    checked_decision = validate(decision)
    observation = execute_if_allowed(checked_decision)
    state = transition(state, checked_decision, observation)

return state.result

The surrounding code must validate each decision, limit each run, record each action, and stop unsafe work.

The application defines the available actions and the maximum number of steps. It also selects the authenticated user and checks the completion condition.

Enforce these controls in code. A prompt cannot enforce them.

Follow one request through the architecture

Consider this request:

I was charged twice for invoice 456. Please refund the extra payment.

The application first creates a run. It copies the authenticated user ID from the session and stores the original request.

The first model call receives the request and a small set of allowed actions. It might return this decision:

{
  "action": "read_invoice",
  "arguments": { "invoice_id": "456" },
  "reason": "The current payments must be checked before a refund can be proposed."
}

That response is a proposal, not permission. The application validates the schema and checks whether the user may read invoice 456.

The invoice service returns two settled payments with the same amount. The application adds a small, structured result to the next model context.

The model can now request a refund proposal. Code calculates the refundable amount and evaluates the approval policy.

request received
  -> model requests invoice read
  -> code checks access
  -> invoice service returns two payments
  -> model requests refund proposal
  -> code calculates amount and checks policy
  -> application waits for approval
  -> code commits one refund
  -> payment service returns a receipt
  -> run completes

Notice which parts the model does not control. It does not choose the authenticated user, calculate money, approve the refund, or declare the payment successful.

The model handles the ambiguous parts. It interprets the request, selects a relevant read, and explains why the evidence supports a proposal.

This division is the central architecture decision. Later parts add better runtime control and richer tools without changing that boundary.

Decide which work belongs in code

A deterministic rule gives the same result for the same verified input. Use code for these rules.

Use the model when the system must interpret unclear language or compare uncertain evidence.

Keep model judgment for tasks such as:

  • interpreting an unclear request.
  • choosing among allowed tools.
  • comparing several pieces of evidence.
  • drafting an explanation.
  • deciding whether more information is needed.

Keep deterministic code for:

  • authentication and authorization.
  • exact calculations.
  • schema validation.
  • rate and spend limits.
  • state transitions.
  • database writes.
  • approval checks.
  • idempotency.
  • audit records.

Consider a refund. A model may interpret why the customer is asking and identify which policy might apply. Code should load the account under the authenticated identity, calculate the amount, decide whether approval is required, and execute the payment once.

A useful design exercise is to make a table with three columns:

Decision Owner Reason
What does the user want? Model Language is ambiguous
May this account read the record? Code Permission must be enforced
Which source answers the question? Model within an allowed set Relevance needs judgment
What is the total? Code Arithmetic should be exact
Did the write succeed? Application code Confirm the result with the service that performed the write

Assign every decision to either application code or the model. Do this before you implement the decision.

Deterministic does not mean infallible

Deterministic code can still contain bugs. The term means that the same verified input follows the same programmed rule.

That property makes the result reproducible and easier to test. It does not prove that the rule is correct.

Model output has a different failure shape. The same request can produce different wording, tool choices, or arguments across runs.

Variation enters through more than sampling:

Source Observable effect Practical control
Sampling and decoding Different text or action choices Fixed settings, schemas, and repeated-trial evals
Model or provider update Behavior changes without application-code changes Pin versions when possible and run regression evals
Context construction Different evidence produces a different decision Version and test the context builder
Tool and API data Live facts change between steps Record source versions and observation times
Concurrency Another worker changes state first Transactions, leases, and optimistic version checks
Time-dependent rules The same request crosses a deadline Use a trusted server clock and record the evaluated time

A deterministic boundary controls how the application responds to this variation. It does not remove variation from the model or the outside world.

Model settings can reduce variation, but they do not create the guarantees of an authorization check or database constraint.

Use structured output to narrow what the model can return:

{
  "action": "read_invoice | propose_refund | ask_user | finish",
  "arguments": {},
  "evidence_ids": ["string"],
  "user_message": "string"
}

Schema validation answers a limited question: does the response have the expected shape?

It does not prove that the selected action is appropriate, the evidence is sufficient, or the user has permission.

Those checks need separate code and tests.

Use several kinds of control together:

Control What it catches What it cannot prove
Output schema Missing fields and invalid types Correct judgment
Allowlist Unknown actions Correct action choice
Authorization Forbidden resource access Factual accuracy
Business rule Invalid amount or state change Good explanation
Step budget Runaway loops Successful completion
Evaluation Repeated behavioral failures Permission to act

Do not make a model prompt responsible for a rule that code can check directly.

Separate the system into layers

A practical agent has four layers.

The model boundary

This layer translates between the application and a model provider. It sends messages, schemas, and tool definitions.

It converts each model response into a defined data structure. It also returns token usage, refusals, and provider errors.

Convert provider-specific formats at this boundary. Other application components can then use one internal format for every provider.

The control loop

This layer owns the run. It stores state, enforces budgets, calls the model, validates decisions, dispatches allowed work, and decides when to stop.

We build this in Part 2.

The capability layer

Tools let the application read data or request actions through APIs, databases, files, search services, and internal services.

Give each tool one narrow operation. Do not give the model unrestricted access to a shell, database, or network client.

We cover direct tools in Part 3 and MCP in Part 4.

The policy and evidence layer

This layer decides what the run may access, what information enters context, what must be approved, and how behavior is evaluated.

Later parts explain how to select context, retrieve evidence, evaluate behavior, and restrict production access.

These layers should communicate through defined data structures. Avoid passing one large conversation object through every component.

For example, the control loop can send a ToolRequest to the capability layer. The capability layer returns a ToolResult with a status and data.

ToolRequest {
  run_id
  tool_name
  arguments
  authenticated_principal
}

ToolResult {
  call_id
  status
  data
  error_code
  receipt_id
}

This boundary makes each layer testable without a live model. It also prevents provider response objects from spreading through the application.

State is part of the architecture

Store run state outside the conversation transcript.

The application needs explicit fields such as:

  • run ID.
  • current status.
  • current step.
  • authenticated principal.
  • verified resource IDs.
  • completed tool calls.
  • remaining budget.
  • pending proposal.
  • approval status.
  • final receipt or result.

Send the model only the state fields that it needs for the current step. Store the complete state outside the model.

The application can then resume after a crash.

Explicit state also makes errors easier to classify. "The answer was wrong" does not identify the failed component.

A trace can show that the lookup succeeded, the policy source was missing, and the model answered without evidence.

State and context serve different purposes. State contains application facts needed to control the run.

Context contains the selected information shown to the model for one decision. The complete state should rarely enter context.

Suppose the run stores a payment-service access token. The tool adapter may need that token, but the model does not.

The context can contain a verified invoice summary while the secret remains outside the model boundary.

Autonomy has a cost

A model can select the correct action in one step and select the wrong action in a later step.

A longer run gives the model more chances to make an error. Measure the complete task instead of inferring reliability from one step.

Before adding a new model decision, ask:

  • Why must the model make this decision?
  • Could code choose the step instead?
  • Which new action sequences can this decision create?
  • Which events will the application record for each sequence?
  • What happens when the model chooses badly?
  • Can the system recover without causing a second side effect?

You can estimate the growth in possible paths without advanced mathematics. If a model can choose five actions for four turns, many sequences become possible.

Most sequences will not make sense. Your limits and state rules must still handle them.

Common architecture failures include:

Failure Cause Design response
Endless lookup loop No enforced stopping rule Set step and tool budgets in code
Action without evidence Completion rule checks only model text Require verified evidence IDs
Cross-account read Identity comes from model arguments Use the authenticated session
Duplicate external action Retry repeats a completed write Use an idempotency key and receipt
False success Model says the write worked Require confirmation from the source service
Unrecoverable crash Progress exists only in chat history Persist explicit run state
Unexplained result Decisions and versions were not recorded Store an ordered trace

These are application failures. A better prompt may improve behavior, but it cannot replace the missing control.

A good first milestone

Keep the first version small. Add memory, RAG, more agents, and more tools only when the task requires them.

A good first milestone has:

  • one real request.
  • one model response with a defined schema.
  • one test model that always returns a known response.
  • one explicit state object.
  • one success case.
  • one refusal or missing-input case.
  • one limit on steps, time, or cost.
  • one event log that shows every decision and tool result.

Build and test this small system before you add more tools.

Its event log will show which decisions the model makes, which rules the application enforces, and which failures need more tests.

Architecture review checklist

Before implementation, review the design with these questions:

  • Is the task specific enough to define success and failure?
  • Could one model call or a fixed workflow complete it?
  • Which decisions need language judgment?
  • Which decisions require deterministic code?
  • What data can the model see?
  • Which identity does each tool use?
  • Which actions change external state?
  • What evidence must exist before each action?
  • Which actions need human approval?
  • What stops the run?
  • Which state survives a process crash?
  • Which event proves that the task completed?

If an answer is unclear, keep the capability out of the first release. Add it after the boundary, failure behavior, and test are explicit.

Next: AI Agents Part 2: Harness and CLI shows how to control each run and inspect each step from a command-line interface.