Output Contracts: Schema Validation Is the Cheapest Agent Reliability Layer

An agent that writes "approximately $50k" into a numeric deal-size field does more damage than an agent that crashes. The crash pages someone. The bad write sits in your CRM for three weeks, feeds a forecast, and gets discovered when a VP asks why pipeline math stopped adding up. By then you cannot tell which of 4,000 records the agent touched are trustworthy.

This is the shape of most real agent incidents. When teams trace production failures back to a root cause, the model reasoned fine. The failure was a malformed handoff between steps that nobody validated: a missing field, a string where a number should be, a hallucinated enum value, a tool response whose shape changed after a vendor update. We covered the broader taxonomy in why AI agents fail in production, and the pattern holds: the seams break more often than the brain.

The fix is old technology wearing a new job title. Schema validation at every boundary, enforced at runtime, with rejection instead of pass-through. Data engineering teams have run this play for years under the name data contracts. Applied to agents, it is the cheapest reliability layer you can ship, and most teams still skip it.

Handoff failures, not reasoning failures

Agent pipelines have a property that single-call LLM apps do not: errors compound across steps. A 95% per-step success rate across a 10-step pipeline yields roughly 60% end-to-end success. When one of those steps emits output that is subtly wrong in shape rather than loudly wrong in behavior, the error does not stop the pipeline. It propagates.

The agent reliability score framework from the platform engineering community makes the same argument from the operations side: before an agent goes live, the platform should guarantee that malformed outputs cannot reach systems of record. That guarantee has to be structural. You cannot prompt your way to it, because the failure mode is precisely that the model occasionally ignores the prompt.

Three examples of what unvalidated handoffs look like in practice:

• Silent field drops. The model returns 9 of 10 expected fields. Your dict.get() calls default to None, the CRM write succeeds, and a customer record loses its renewal date.
• Type drift. The model returns "priority": "high" on Monday and "priority": 1 on Thursday. Downstream code that compares against strings silently misroutes every Thursday ticket.
• Schema drift from tools. A vendor API adds a field or renames one, the tool result no longer matches what your prompt describes, and the model starts confabulating the missing values instead of erroring.

None of these are intelligence problems. All of them are contract problems, and contracts are checkable by machines in microseconds.

The four boundaries that need a contract

An output contract is a schema plus an enforcement point. The schema says what valid data looks like. The enforcement point is where non-conforming data gets stopped. In an agent system there are four boundaries, and every production incident we have traced to malformed data crossed one of them unchecked:

Most teams that validate at all only validate the final action. That catches the last error and none of the upstream ones, which means by the time validation fires you have burned the whole pipeline's latency and token cost producing garbage. Validating tool outputs is the one people skip most, on the theory that APIs are deterministic. APIs are deterministic until the vendor ships a change, and your agent is the only consumer that responds to a renamed field by inventing a plausible value.

Defining the schemas is not extra work you would otherwise avoid. Providers already require it at the tool-input boundary: Anthropic's tool use API takes a JSON Schema input_schema per tool, and OpenAI's structured outputs accept a schema with a strict flag that constrains generation to conform. The marginal work is extending the same discipline to the other three boundaries, where nobody forces you.

In Python the standard pattern is to define each boundary as a Pydantic model and export JSON Schema from it, so one definition serves as the provider-facing schema, the runtime validator, and the documentation:

class TicketTriage(BaseModel):
    ticket_id: str = Field(pattern=r"^TKT-\d{6}$")
    priority: Literal["p1", "p2", "p3"]
    route_to: Literal["billing", "technical", "account"]
    confidence: float = Field(ge=0.0, le=1.0)
    summary: str = Field(min_length=20, max_length=500)

result = TicketTriage.model_validate_json(raw_output)  # raises on violation

Five fields, four of them constrained beyond bare types. The Literal on route_to is the line that prevents the model from inventing a "customer-success" queue that does not exist and having the write silently succeed against a permissive API.

The enforcement ladder

Validation is not one technique. There are three rungs, in increasing cost and strength, and the right answer for most boundaries is not the strongest one.

Rung one, parse checks. Does the output parse as JSON at all? This is free and catches the crudest failures: markdown fences around the payload, truncation, prose apologies instead of data. If you do nothing else, do this. But a parse check alone is how "approximately $50k" gets through, because it is valid JSON.

Rung two, schema validation. Validate the parsed object against the full contract: types, required fields, enums, ranges, formats. Pydantic v2 runs its core in Rust and validates typical agent payloads in tens of microseconds, which rounds to zero next to a multi-second model call. This rung catches the large majority of malformed handoffs and is the correct default at all four boundaries.

Rung three, constrained decoding. Instead of validating after generation, constrain generation itself so invalid output cannot be produced. Libraries like Outlines compile a JSON Schema or regex into a token-level automaton that masks invalid tokens at each decoding step, and vLLM ships this as structured outputs with multiple grammar backends for open-weight serving. OpenAI's strict mode is the same idea run provider-side, and it took their schema-conformance eval from under 40% with prompting alone to 100%.

The instinct is to jump straight to rung three everywhere. Resist it. Constrained decoding guarantees shape, not sense. A constrained model will always emit a valid enum member, including when the honest answer is "none of these apply," at which point you have converted a detectable failure into an undetectable one. Constraints also cannot express cross-field rules (refund amount must not exceed original charge) or anything requiring a lookup (this customer ID must exist). Rung two handles those with custom validators. Use rung three where retry latency genuinely hurts or where you control the serving stack and the schema is stable; use rung two everywhere, including on top of rung three.

Fail closed, retry typed

A contract without a rejection path is documentation. The enforcement policy that makes contracts pay for themselves has three parts.

First, fail closed. Output that violates the contract does not proceed, full stop. No "log a warning and continue," because a warning nobody reads plus a bad CRM write is just a bad CRM write with an alibi. This matters most at the final-action boundary; the writeback patterns post goes deeper on why writes into systems of record deserve the strictest gate in the pipeline.

Second, retry with the error, not from scratch. A blind retry re-rolls the dice with the same odds. A typed retry appends the exact validation failure to the conversation:

Your previous response failed validation: route_to must be one of billing, technical, account; received customer-success. Return a corrected response.

In our builds, one typed retry resolves the large majority of schema violations, because the model was not incapable of conforming, it just did not on that sample. Cap retries at two or three, then route to a human queue or a safe fallback. Unbounded retry loops are their own incident category.

Third, log every rejection as a structured event: boundary, schema version, field, violation, retry count, resolution. This is the quiet compounding benefit. Nondeterministic bugs ("the agent sometimes messes up tickets") become deterministic, queryable data ("route_to violations tripled after the prompt change on the 14th"). Your rejection log turns into a free eval set of exactly the cases your model finds hard, which is worth more than any synthetic benchmark of your pipeline.

The reasoning-quality objection

The strongest argument against strict contracts is that forcing structure degrades thinking. It deserves a straight answer, because there is real evidence on both sides.

The "Let Me Speak Freely?" study (Tam et al., 2024) found that format restrictions measurably hurt performance on reasoning-heavy tasks, with stricter constraints causing larger drops. The team behind Outlines ran a detailed rebuttal showing that with well-designed schemas and prompts, constrained generation matched or beat unconstrained generation on the same benchmarks, and argued the original degradation came from awkward schema design rather than constraint itself.

The practical synthesis is boring and works. Give the model room to think before it commits: put a free-text reasoning field first in the schema, then the structured decision fields. The model reasons in prose, then fills the contract, and field ordering in the schema controls generation order. You keep the guarantee where it matters (the fields machines consume) without squeezing the deliberation that produces good values for them. If you use provider-side reasoning or extended thinking, the same separation happens for free.

The counterintuitive part is where contracts sit in the reliability budget. Teams spend heavily on evals and observability, which measure and detect, before spending anything on contracts, which prevent. Evals tell you your agent is wrong 8% of the time. Contracts stop a specific class of those wrongs from reaching production at all, cost microseconds per check, and require no labeled data. On effort-to-incidents-prevented, boundary validation beats every other line item in the stack, which is why it should be the first layer you ship, not the last.

Sequencing contracts into a real build

Where this shows up in delivery work: when OpenNash audits an existing agent system, the first artifact we produce is a boundary map - every point where model output crosses into a tool, another step, or a system of record, annotated with whether a contract exists and what enforces it. On most systems that have been live for a few months, fewer than a third of boundaries have any check beyond a parse. That map alone usually explains the incidents the team could not reproduce.

The build sequence that follows is deliberately incremental. Contracts on the final-action boundaries first, because that is where silent damage lands in customer-visible systems. Then tool outputs, because vendor drift is the failure you cannot prevent, only catch. Then step handoffs, then constrained decoding on the hot paths where retry latency costs real money. Each layer ships with rejection logging wired into the client's existing observability, so the ownership handoff includes not just the validators but the dashboard that shows what they are catching.

If you are running an agent in production and cannot say which of its boundaries are validated, that boundary map is the conversation to have. Bring your pipeline diagram to a working session with OpenNash and leave with the map, the priority order, and the first two schemas drafted.

Start with one boundary this week

You do not need a platform migration to start. Pick the single agent write that would hurt most if it went bad silently - usually a CRM, helpdesk, or ERP writeback. Define its Pydantic model with real constraints, not just types: enums for anything categorical, ranges for anything numeric, patterns for anything with an ID format. Wire it to fail closed with one typed retry and a log line. Ship it, then read the rejection log after a week.

The log will tell you two things: how often your agent was producing malformed output that you were previously consuming as if it were fine, and which boundary deserves a contract next. Most teams find the first number uncomfortable. That discomfort is the cheapest reliability lesson available, because every rejection in that log is an incident that did not happen.