Spec-Driven AI Workflows: Why Pilots Fail Before the Agent Is Built

A mid-market services company wants an AI agent to handle inbound quote requests. The demo is easy. Feed it three clean emails, give it a product sheet, ask for a draft reply, and it looks ready.

Then the real queue arrives. One prospect sends an expired price list as an attachment. Another asks for a discount that only applies to renewals. A third asks for a regulated configuration that sales is not allowed to quote without legal approval. A fourth is already an active customer under a master services agreement, but the name in the email does not match the CRM account.

The pilot did not fail because the model was dumb. It failed because nobody wrote down the job.

That is the operating lesson behind Erik Hanchett's AWS session, Using Spec-Driven Development for Production Workflows. The software pattern applies cleanly to business AI: when a system is going to perform real work, the specification has to come before the agent.

For operators, that means changing the first question. Not "which model should we use?" Not "can we get a chatbot on the site by Friday?" The first question is: what exact workflow are we asking this system to perform, and how will we know it did the work correctly?

The Pilot Fails in the Blank Space

Most AI pilots start with a sentence that sounds clear enough in a meeting:

"Automate Tier 2 support."

"Draft RFP responses."

"Summarize contracts."

"Update CRM records from emails."

Those are not workflow specs. They are areas of interest. They do not define the task boundary, the input shape, the allowed tools, the source of truth, the review path, or the success criteria. A capable model will fill those gaps with assumptions because that is what the system is built to do. In a demo, those assumptions are invisible. In production, they become incidents.

Hanchett uses the phrase "AI intern," and the analogy is useful. A junior person can do good work when the assignment is scoped, reviewed, and attached to a clear standard. The same person can cause chaos if every hallway comment becomes an urgent task. AI agents are similar. Give them a vague business goal and broad tool access, and they will confidently improvise across gaps the business never agreed to.

The enterprise data points say the same thing. McKinsey's 2025 State of AI survey found that 88 percent of organizations report regular AI use in at least one business function, but only about one-third have begun scaling AI across the enterprise. The same report notes that high performers are much more likely to redesign workflows and define when model outputs need human validation. Access to AI is common. Operational integration is scarce.

That gap is where pilots die.

What Spec-Driven Development Gets Right

AWS's Kiro project is a useful reference because it turns a vague AI coding request into a structured set of artifacts. The Kiro specs documentation describes three files: requirements.md, design.md, and tasks.md. The requirements define behavior and acceptance criteria. The design explains architecture and implementation choices. The tasks break the work into trackable steps.

Business teams do not need to adopt Kiro to use the pattern.

Kiro's feature spec docs push requirements into a testable form using EARS-style acceptance criteria, such as WHEN [condition/event] THE SYSTEM SHALL [expected behavior]. That format is useful because it forces the writer to name the trigger and the expected response. It turns "handle refund requests" into something closer to:

WHEN the refund amount is above $500, THE AGENT SHALL route the case to a human approver and SHALL NOT issue the refund directly.

That is not elegant prose. Good. It is a rule someone can test.

Kiro also has requirement analysis for inconsistencies, ambiguities, conflicting constraints, unstated assumptions, and missing edge cases. Those are exactly the failure modes that show up in business pilots. "Large customer" is ambiguous. "Fast response" is ambiguous. "Use the latest policy" is ambiguous unless the system of record is named. "Escalate sensitive cases" is ambiguous unless sensitive is defined.

Spec-driven development works because it turns ambiguity into reviewable artifacts before implementation begins.

Translate the Pattern From Code to Operations

A production AI workflow needs its own version of requirements.md, design.md, and tasks.md.

The workflow requirements document should answer what the agent is allowed to do. It includes the task boundary, user roles, accepted inputs, required outputs, business rules, allowed decisions, forbidden actions, and human approval triggers. For a quote desk agent, this is where you state that the agent may draft quotes but may not approve non-standard discounts, may not create a new legal term, and must cite the price book version it used.

The workflow design document should answer how the agent will do the work. It names the systems of record, tools, retrieval sources, permission model, event triggers, output schemas, escalation channels, observability points, and rollback paths. This is where vague architecture becomes operational. Does the agent read Salesforce directly or through a filtered API? Does it write to the CRM or produce a draft task for a human? What happens if the price book and ERP disagree?

The workflow task plan should answer what gets built and tested. It converts the design into implementation steps: connect the CRM read tool, create the quote draft schema, add discount policy retrieval, build the approval queue, write the eval cases, instrument traces, and define the release gate.

This is the same operating pattern behind workflow discovery before AI automation and workflow pass/fail criteria. The agent should not be the first artifact. The workflow spec should.

The Spec Is Also the Eval Plan

OpenAI's guidance on evaluating agent workflows is direct: use traces, graders, datasets, and eval runs to improve agent quality. A trace captures model calls, tool calls, guardrails, and handoffs. Graders score behavior against structured criteria. Datasets let teams compare changes across representative cases.

None of that works without a spec.

If the spec says the agent must route every quote above a margin threshold to finance, the eval can check whether that handoff happened. If the spec says the agent may only use the current price book, the eval can check source versioning. If the spec says the output must include a confidence reason and a cited policy, the eval can inspect the final JSON.

Without a spec, evaluation turns into taste. Someone reads an output and says it feels right or wrong. That may be useful during exploration, but it is not a release gate. A release gate needs criteria that are stable enough to run every time the prompt changes, the model changes, a tool changes, or a policy changes.

This is why every spec should include a golden set of representative cases before the agent launches:

That list becomes the first eval suite. After launch, every new failure becomes another case, which is the feedback loop described in Every Production Failure Should Become an Eval Case.

The Spec Controls Tool Access

Anthropic's Building Effective Agents makes a distinction that matters for operators: workflows follow predefined code paths, while agents dynamically direct their own process and tool use. It also recommends the simplest solution that works, because agentic systems often trade latency and cost for better task performance.

That advice should shape the workflow spec. Do not give an agent broad tools because the demo might need them. Give it the smallest set of actions required by the defined job.

For a support workflow, the spec might say:

• The agent may read tickets, account status, order status, and published policy.
• The agent may draft customer replies and internal notes.
• The agent may tag a ticket using approved categories.
• The agent may not issue refunds above $500.
• The agent may not modify billing records.
• The agent must escalate cases involving legal threats, regulated data, or abusive content.

Those are not prompt suggestions. They are access-control requirements. Some should live in the system prompt. Some should live in tool wrappers. Some should live in deterministic code. Some should live in human review.

Kiro's steering docs are another useful analogy. Steering files give the coding agent persistent knowledge about a workspace: product goals, technology choices, structure, standards, and security rules. Business agents need the same kind of persistent operating knowledge, but aimed at the workflow: policy rules, approval thresholds, tone standards, escalation paths, data boundaries, and audit obligations.

The spec decides what the agent knows all the time, what it retrieves only when needed, and what it is never allowed to decide alone.

The Spec Makes Governance Practical

Governance becomes painful when it is bolted on after a pilot succeeds. By then, the demo has already shaped expectations, the agent has already been wired to tools, and the business sponsor is already asking why legal, IT, or finance is slowing things down.

The better move is to write governance into the workflow spec.

NIST's AI Risk Management Framework is voluntary and sector-neutral, but the operating idea is relevant to mid-market teams: risk management should be incorporated into the design, development, use, and evaluation of AI systems. That does not mean every support bot needs a 60-page risk document. It means the spec should answer practical risk questions before launch:

• What business damage can a wrong action cause?
• Which actions are reversible?
• Which data can the agent see?
• Which data can the agent write?
• Which decisions require human approval?
• Which outputs need citations?
• Which events must be logged?
• Who reviews exceptions each week?

For private-equity operators, this matters because an AI workflow is not just a technology asset. It is an operating asset. If a portfolio company can show the workflow spec, eval cases, audit logs, approval gates, and production metrics, the AI program becomes easier to diligence and easier to improve. If the only artifact is a prompt in a vendor console, the value is fragile.

A Practical Spec Template

OpenNash uses a compact spec for first workflows. It is intentionally plain because the goal is alignment, not documentation theater.

1. Workflow name and owner

Name the workflow and the accountable operator. Shared ownership usually means no ownership.

2. Business outcome

Define the operational metric: cycle time reduction, deflection rate, quote turnaround, invoice exception rate, backlog reduction, or first-contact resolution.

3. Scope

Write what is in scope and out of scope. A good first agent has a narrow job.

4. Inputs

List the exact inputs: emails, tickets, PDFs, CRM fields, forms, call transcripts, knowledge base articles, ERP records, or uploaded documents.

5. Systems of record

Name the source of truth for each fact. If price comes from ERP and customer status comes from CRM, write that down.

6. Outputs

Define the output schema: draft reply, JSON decision, CRM task, quote draft, exception memo, approval request, or dashboard event.

7. Allowed tools and actions

List read tools, write tools, side effects, rate limits, and forbidden actions.

8. Decision rules

Write the business logic in testable form. Use WHEN...THE AGENT SHALL... where possible.

9. Human review

Define approval gates, escalation rules, reviewer roles, and service-level expectations.

10. Eval cases and release bar

Create representative cases before launch. Set the pass rate, hard-fail rules, latency target, cost target, and required trace fields.

11. Production monitoring

Define what gets logged, what gets reviewed weekly, what triggers rollback, and how failures become evals.

This template is not heavy. It is the minimum contract required before an agent touches live work.

Where OpenNash Fits

The hardest part of agent deployment is not getting a model to respond. It is turning a messy business process into a clear operating contract that an agent, a human reviewer, an engineer, and an auditor can all understand.

That is the OpenNash lane.

We start by mapping the workflow and finding the proof source. We write the spec before the build. We turn real historical cases into evals. We separate deterministic steps from model judgment. We design approval gates where the downside is too high. We instrument traces so failures can be replayed. Then we build the agent around that operating model rather than hoping a prompt will hold the whole thing together.

The spec also keeps the buyer honest. Sometimes the right answer is not a custom agent yet. Sometimes the source data is too messy, the workflow is too open-ended, or the review cost is as high as doing the work manually. A spec exposes that before the budget is gone.

The useful AI programs will not be the ones with the most pilots. They will be the ones that turn real workflows into specified, measured, governed systems of work.

Before you build the agent, write the job. Then build the agent that can prove it did that job.