Harbor Framework: Agent Evals for Real Business Work

A customer support agent looks great in a demo. It answers three sample questions, cites a help article, and opens a ticket. Then the real queue arrives: one customer asks for a refund and a shipping update in the same message, another has an account merge problem, a third is in a jurisdiction with a different policy, and the fourth triggers a backend error that the agent has to notice rather than talk around.

Harbor lives in that gap.

Harbor describes itself as a framework for evaluating agents in sandboxed environments. The plain English version is more useful for buyers: Harbor gives teams a way to turn agent work into repeatable tests. Define the task, spin up the environment, run the agent, verify the result, score the attempt, and compare runs across models, prompts, tools, or agent implementations.

For companies trying to move AI agents from pilot to production, this is not a technical side quest. It is the question that decides whether the project can touch real operations.

The Business Problem: Demos Do Not Prove Operations

Most AI agent projects fail the same way. The first prototype works on a clean example. The buyer sees enough to believe the model is capable. The project then hits the messy operational layer that was missing from the demo:

• The agent needs a database, browser, API, file system, or terminal.
• The work has hidden state, permissions, retries, and partial failure.
• Correctness depends on business rules, not just plausible text.
• One prompt fix can break another case.
• A model upgrade changes behavior without warning.
• No one can compare last week's agent to this week's agent on the same work.

Production AI needs an eval harness, not just a chat window. A harness answers four business questions:

1. Can the agent complete the workflow from start to finish?
2. Did it complete the work according to our rules?
3. Did the latest change make the system better or worse?
4. Can we prove the answer to customers, auditors, operators, and ourselves?

Without that harness, deployment decisions lean on demos and confidence. With it, the company can make agent changes the way engineering teams make software changes: run the tests, inspect failures, decide whether to ship.

What Harbor Actually Gives You

The Harbor core concepts map cleanly to the way real agent work should be tested.

A task is one instruction, one container environment, and one verifier. For a business workflow, that could mean "resolve this refund ticket," "reconcile this invoice," "extract these contract obligations," or "update this CRM account from a messy email thread." The task includes the environment the agent needs and the test script that decides whether the work passed.

A dataset is a collection of tasks. In business terms, that is your golden set: the real cases that represent the queue you care about. Good datasets include boring happy paths, strange exceptions, policy edge cases, permission boundaries, and past production failures.

An agent is the program trying to complete the work. Harbor supports agent interfaces and pre-integrated CLI agents, which matters because companies do not want their evals tied to one vendor. You should be able to test a coding agent, support agent, internal operations agent, or custom OpenNash-built agent against the same cases.

A container environment is the sandbox where the work happens. Harbor supports Docker-style environments and cloud sandbox providers such as Daytona, Modal, E2B, Runloop, Tensorlake, LangSmith, Blaxel, Novita Sandbox, EC2, and Beam. The business value is isolation and repeatability. The agent can install packages, call tools, edit files, and make mistakes without touching production systems.

A trial is one attempt. A job is a collection of trials, often across many tasks, agents, and models. That lets a team ask useful questions: does the new model improve Tier 2 ticket resolution, does the cheaper model still pass policy checks, does the new prompt reduce tool errors, and which failures remain after the last fix?

The getting started docs also make the operating shape clear: harbor run is the primary command for running evals or generating rollouts, registered datasets can be pulled automatically, local datasets can be run from a directory, and cloud sandboxes can run many trials concurrently. That concurrency matters because sandboxed agent evaluations are slow by nature. Agents take many steps, and each step can include tool calls, command execution, or file operations.

Harbor packages the eval work that serious teams were already stitching together with custom scripts, spreadsheets, and shared run folders.

Why Containers Matter for Agent Evals

For simple question answering, a spreadsheet of prompts and expected answers may be enough. For agents, it breaks quickly.

Agents change state. They create files, call APIs, run commands, update records, and recover from errors. You cannot test that kind of behavior with a static prompt list. You need an environment the agent can act inside and a verifier that can inspect the result.

Harbor's container model matters because agent work changes state. A support agent can be tested against a fake ticketing API. A finance agent can reconcile CSVs, PDFs, and ledger fixtures inside a sandbox. A sales operations agent can edit a CRM-like database and then be graded on whether the right fields changed once, and only once. A coding agent can fix a broken repo and run the test suite.

The verifier is where business rules become enforceable. Harbor's task structure docs describe tests that produce reward files, including simple pass/fail rewards or JSON metrics. That maps directly to workflow quality:

The question changes from "did the agent sound good?" to "did the agent do the job?"

Harbor Also Solves the Evidence Problem

Evaluation is only half the business problem. The other half is evidence.

If an agent project is important, someone eventually asks for proof: a customer, compliance lead, board member, operations manager, or engineering owner. They want to know what changed, how it performed, what failed, and whether the team can reproduce the result.

Harbor's recent product direction points at that need. In May 2026, Harbor announced job result sharing through Harbor Hub, so teams can upload or stream job results and share them with users or organizations instead of zipping run folders around. That sounds small until you have watched an AI pilot die because no one can find the exact run that justified a deployment.

The separate verifier sandbox feature points at the same problem from a security and reliability angle. Verification can run in a different sandbox from the agent, with explicit artifact handoff. For business teams, this matters because the grader should not always live in the same environment the agent can modify. Some checks need clean dependencies, hidden grading code, or a stricter security boundary.

Harbor's multi-step task support matters for long-running work. Real workflows are rarely one move. A Tier 2 support case might classify the issue, retrieve policy, inspect order state, draft a reply, request approval, then update the ticket. Multi-step tasks let each stage have its own instruction and verifier, which is a closer match for how operations actually run.

The business result is a trail of evidence:

• Which agent ran.
• Which model and prompt version were used.
• Which task and dataset were tested.
• Which environment was created.
• Which artifacts were produced.
• Which verifier checked the result.
• Which reward or metrics came back.
• Which results are worth sharing with the team.

That evidence is what turns "we think it works" into "we can show where it works, where it fails, and what changed."

Where OpenNash Fits

Harbor is a framework. It does not, by itself, know your refund policy, support tiers, quote desk, invoice rules, compliance boundaries, or CRM fields. OpenNash brings that workflow knowledge into the harness.

OpenNash builds production AI agents around the operating layer that Harbor makes easier to test:

1. We map the workflow before building the agent.
2. We pull real cases from tickets, invoices, forms, documents, calls, chats, or CRM activity.
3. We turn those cases into a golden dataset with expected outcomes.
4. We write deterministic verifiers for the rules that should never be judged by taste.
5. We add model-based or human review only where judgment is actually required.
6. We connect eval runs to CI, release gates, dashboards, and weekly failure review.
7. We hand over the agent, traces, evals, and operating docs so the client owns the system.

Harbor can become part of that owned evaluation layer. For a client with complex tool use, local software environments, or multi-step work, we can package the workflow as Harbor tasks and datasets. For a customer support deployment, that might mean a Harbor dataset of historical tickets with mocked backend APIs and verifiers for routing, policy compliance, citation quality, and write-back behavior. For finance operations, it might mean container tasks with invoices, ledger fixtures, and approval thresholds.

The agent itself may be OpenNash CX, a custom back-office agent, a coding assistant, or a client's existing agent stack. The test harness should be model-agnostic. If a client wants to compare GPT, Claude, Gemini, an open-weight model, or a cheaper routing path, the same Harbor-style cases can score each option.

That beats asking which model is smartest. The better question is: which agent clears our workflow's release bar at the lowest operating risk and cost?

How a Harbor-Style OpenNash Engagement Would Work

The first week should be evidence collection, not prompt writing.

We start with the queue the business wants to automate. For customer support, that might be 100 resolved Tier 1 and Tier 2 cases. For finance, it might be 80 invoices with normal, disputed, and exception outcomes. For sales operations, it might be 50 account updates and quote requests. The goal is to capture the shape of the work, not a sanitized sample.

Then we define the reward. Some checks are hard assertions: the total must match, the policy threshold must be respected, the CRM update must target the right record. Some checks are rubric-based: the support reply must answer all customer questions and use the right tone. Some cases require escalation, and the correct answer is to stop.

Next comes the harness. We create the task directories, fixtures, environment definitions, and verifier scripts. We decide whether the verifier should share the agent environment or run separately. We decide what artifacts need to be copied out: answer JSON, ticket draft, tool trace, API log, screenshots, or generated files.

After that, we run baseline jobs. The buyer gets the first real picture of agent readiness: cases passed, cases failed, failure types, latency, cost, and what the agent did in the environment.

Only then do we improve the agent. Prompt changes, tool changes, retrieval changes, model routing, permission gates, and deterministic code all get tested against the same dataset. Every production failure that appears later becomes a new case. The suite grows from reality.

What Buyers Should Take From Harbor

The takeaway is not "every company should install Harbor tomorrow." The agent market is moving from demos to verifiable work.

If you are buying, building, or approving AI agents, ask vendors these questions:

• What is the task format for our workflow?
• What environment does the agent act inside during testing?
• What verifier decides whether the work passed?
• What dataset represents our real cases?
• What happens when a production failure occurs?
• Can we compare two models or prompts on the same cases?
• Can we share the job evidence with operators and decision makers?
• Who owns the eval suite after launch?

Harbor gives teams a practical vocabulary for those questions. OpenNash turns that vocabulary into an operating model around your actual work.

For low-risk chatbots, this may be more structure than you need. For agents that touch refunds, records, documents, code, support queues, finance operations, or regulated workflows, it is the difference between hoping an agent works and running a system that can prove it.

The teams that win with AI automation will not be the ones with the most polished demo. They will be the ones with clear tasks, strong verifiers, a useful failure loop, and the discipline to measure every change before it reaches customers.

Harbor is solving the proof problem. OpenNash builds that proof layer into production workflows.