A refund agent has twelve rules. It must verify identity, check the order, use the current return window, apply a customer-tier exception, stay under a dollar limit, avoid internal notes, ask for approval in two special cases, and call tools in the right order. The final reply sounds perfect. The problem is that it issued the refund before checking whether the order had already been refunded.
This is why "we put it in the system prompt" is not a control.
AGENTIF, a NeurIPS 2025 benchmark, collected 707 annotated instructions from 50 agent applications. The instructions averaged 1,723 words and 11.9 constraints. Even the best-performing model perfectly followed fewer than 30 percent of the complete instructions. Models struggled most with complex conditional structures and tool specifications, and performance degraded as instructions grew longer. The benchmark does not prove that every twelve-rule agent will fail. It shows why teams need both per-constraint compliance and an all-constraints pass rate.
The right production design gives language to the model and authority to the surrounding system.
A Prompt Is Guidance, Not an Enforcement Boundary
System prompts are useful. They define the agent's role, explain company terms, set tone, describe tools, and tell the model how to handle ambiguity. They are also probabilistic instructions interpreted alongside a long conversation, retrieved documents, tool results, and other rules.
Policy enforcement has a different job. It must answer questions such as:
- May this user read this record?
- Is a $750 refund allowed without approval?
- Can this attachment be sent to that domain?
- Has identity verification already completed?
- Is the requested action available in this workflow state?
- Does the account have a legal hold?
Those questions have data, identity, and state behind them. A model may help assemble the request, but it should not be the final authority.
The distinction becomes clear at the tool boundary:
| Layer | Good use | Bad use |
|---|---|---|
| Prompt | "Explain a denial clearly and offer the approved next step" | "Never refund more than $500" as the only limit |
| Tool schema | Require order ID, reason code, amount, and currency | Accept one free-text request argument |
| Policy decision | Check actor, action, resource, amount, region, and account state | Ask the model whether the action seems permitted |
| Workflow state | Require identity check before refund proposal | Let the model remember whether verification happened |
| Database or API permission | Restrict the credential to allowed rows and actions | Give a shared administrator token and rely on the prompt |
| Human approval | Commit a high-impact exception | Ask a reviewer to approve every harmless lookup |
OWASP calls the broader failure excessive agency: too much functionality, permission, or autonomy. Its direct recommendation is complete mediation, meaning downstream systems validate every request instead of relying on the LLM to decide whether an action is allowed.
This is not distrust of models. It is ordinary systems engineering. Web applications do not ask the page copy to enforce authorization. AI applications should not ask the system prompt to do it either.
Turn the System Prompt Into a Constraint Registry
Most teams cannot say how many requirements their agent has. Rules are scattered across a prompt, tool descriptions, wiki pages, support macros, code, and reviewer habits. Before adding guardrails, build a constraint registry.
Read the full instruction set and capture each requirement as one row:
| Field | Example |
|---|---|
| Constraint ID | REFUND-017 |
| Rule | Refunds above $500 require manager approval |
| Type | Amount, conditional, approval |
| Source | Returns Policy v8, section 4.2 |
| Inputs | Amount, currency, customer tier, order status |
| Enforcement point | Refund API gateway |
| Failure behavior | Deny and create approval request |
| Test owner | Payments QA |
| Policy version | returns-2026-07 |
Classify each constraint into five buckets.
Presentation constraints cover tone, format, required sections, and plain-language explanations. Prompts and output validators handle these well.
Semantic constraints require interpretation: whether a message is threatening, whether a request is a complaint, or whether two clauses conflict. The model can propose a label, with human review where the consequence is high.
Conditional business rules use known facts and thresholds. "If the account is enterprise and the credit exceeds $500, request approval" belongs in workflow code or a policy engine.
Tool and sequence constraints define required parameters, allowed operations, and state transitions. Tool schemas, preconditions, and postconditions should enforce them.
Permission constraints define who may act on which resource. These belong in the downstream identity and authorization layer, using the current user or service principal.
The register exposes contradictions early. One policy may say VIP customers can receive a $750 credit while another caps every automated credit at $500. A longer prompt does not resolve that. Give each policy a precedence, owner, effective date, and explicit conflict rule.
The surprising result is often that the prompt gets shorter. Once hard boundaries move into code, the model receives a focused job instead of a miniature employee handbook.
Put Each Rule at the Narrowest Enforceable Point
Do not build one giant "guardrail model" in front of every action. Enforce a rule as close as possible to the resource it protects.
For authorization, use a policy decision point that receives a principal, action, resource, and context. AWS describes this pattern in Verified Permissions: the application asks whether an identified principal may perform an action on a resource, then enforces the allow or deny decision. Open Policy Agent and custom rules services can serve the same architectural role.
Open Policy Agent provides a general policy engine that separates policy decisions from enforcement. The exact product is secondary. What matters is that policy inputs are structured, decisions are repeatable, and the tool or downstream service refuses the operation when the decision is not allow. A model-generated sentence saying an action is approved is never an authorization token.
For a proposed refund, the request might contain:
{
"principal": "support-agent:on-behalf-of:user-1842",
"action": "refund",
"resource": "order-92117",
"context": {
"amount_usd": 750,
"identity_verified": true,
"already_refunded": false,
"customer_tier": "enterprise"
}
}
The agent can produce this structured proposal. A deterministic service checks policy and returns allow, deny, or approval_required. The refund API refuses to execute without a valid decision tied to the same principal, resource, amount, and policy version. This also limits the damage from prompt injection, because text in a document cannot mint an approval or change the policy result.
Use the narrowest tool that completes the job. issue_refund(order_id, amount, reason) is safer than run_sql(query) or call_api(method, url, body). Narrow tools reduce argument ambiguity and create a clear place for validation, authorization, idempotency, and logging.
Give read and write operations separate credentials. Execute actions in the user's context when the work is performed for a user. Set absolute limits even for approved workflows. A manager approval may permit a $750 refund; it should not turn the agent's credential into an unlimited payments key.
Add both preconditions and postconditions:
- Precondition: order exists, identity is verified, amount is positive, currency matches, refund has not occurred, policy decision allows it.
- Postcondition: one refund record exists, amount matches, ledger state changed once, customer notification is queued, audit event was written.
Preconditions block unsafe proposals. Postconditions catch partial failure and silent side effects.
Test Rules Individually, at Boundaries, and in Conflict
A conversation-level score hides too much. An agent can reach the correct final response after calling a forbidden tool, exposing internal data, or making a state change it later reverses. Test the trace and the resulting system state.
For every constraint, create four kinds of cases:
- Positive: a normal allowed request should succeed.
- Negative: a clearly prohibited request should be denied.
- Boundary: values immediately below, at, and above a threshold.
- Conflict: two applicable rules point toward different outcomes.
Then combine constraints. An enterprise customer requesting $750 within the return window may be allowed with approval. The same amount on an already-refunded order must be denied, not approved. Conditional interactions are where long instructions break.
Measure compliance per rule:
| Metric | What it reveals |
|---|---|
| Constraint pass rate | Which rule the agent or system missed |
| Unauthorized tool proposal rate | Whether the model attempts forbidden actions |
| Enforcement escape rate | Whether any forbidden proposal reached the resource |
| False-denial rate | Whether controls block legitimate work |
| Approval routing accuracy | Whether exceptions reach the right owner |
| Denial explanation quality | Whether users can recover without guessing |
| Postcondition pass rate | Whether the business state is correct after action |
The enforcement escape rate should be zero for high-impact actions. The model may still propose a bad call during testing. The policy layer must block it every time.
Test instruction changes and policy changes independently. A copy edit should not silently alter a refund limit. A policy release should not require changing model wording in the same commit. Version the prompt, model, tool schemas, policy bundle, and workflow graph separately, then record all five in each trace.
NIST's AI RMF Core calls for objective, repeatable, or scalable testing, evaluation, verification, and validation processes with documented results. A constraint registry and executable policy suite turn that principle into a release mechanism.
Design Denials and Escalations as Product Behavior
A guardrail that says only "no" creates support tickets and workarounds. Every denial should tell the workflow what happens next.
Use four outcomes:
- Allow: execute and record the decision.
- Allow with sampling: execute, then add the case to a review sample.
- Require approval: pause with a compact decision packet for the named reviewer.
- Deny and route: block the action, explain the rule, and offer an allowed path.
The agent is useful here. It can translate a policy decision into a clear explanation, gather missing evidence, or draft an approval request. It should not override the decision.
A good approval packet contains the proposed action, affected resource, boundary crossed, evidence, policy citation, available choices, and expiry. It should take under a minute to review. Our guide to approval gates and exception routing covers where to place these checks without training reviewers to rubber-stamp everything.
Make failures observable. Log the proposed tool call, normalized arguments, authorization input, determining policy, decision, approver, final side effect, and correlation ID. For sensitive data, log identifiers and hashes rather than copying the content into every trace.
Run policy in shadow mode before enforcement when replacing a mature workflow. Compare what the new policy would allow or deny against real traffic without changing outcomes. Review disagreements, fix missing context, then turn enforcement on by action class. Do not leave high-risk controls in shadow mode indefinitely.
Finally, maintain a stop switch at the capability level. If refund behavior regresses, disable issue_refund without taking the entire support agent offline. Scoped tools make that possible.
How OpenNash Can Help
OpenNash helps teams replace prompt-only rules with an enforceable agent control plane. We start by extracting the hidden requirements from prompts, SOPs, tool descriptions, and reviewer decisions. Together with policy owners, we turn them into a constraint registry and decide which rules belong in prompts, validators, workflow state, authorization, or human approval.
We then implement the narrow tool contracts, policy decision points, identity propagation, approval paths, audit events, and constraint-level evals. The deliverable includes the denial behavior and rollback plan, not just the happy path. Existing platforms often handle straightforward access control well. Custom work becomes useful when decisions combine company-specific thresholds, workflow state, tenant rules, and several systems of record.
A managed policy service is useful when many applications share rules and audit requirements. Stable local constraints may be clearer in application code. If policy owners cannot agree on the rule, defer the action rather than asking a model to settle an unresolved operating decision.
Book a call to turn one agent prompt into a constraint map and release test suite. Bring the prompt, the tool list, and three decisions that still make reviewers nervous.