Prompt Injection in AI Agents: An Enterprise Security Playbook

An accounts-payable agent receives a routine request: summarize today's invoices and flag duplicates. One PDF contains white-on-white text telling the agent to ignore its task, find recent payroll records, and email them to an outside address. The employee cannot see the instruction. The model can. If that model also has email and file permissions, a document has become a command channel.

That is indirect prompt injection, and it becomes more dangerous as an agent becomes more useful. A chatbot can produce a bad answer. An agent can search private systems, change records, send messages, or move money. The security question is not whether a model will always spot malicious text. It will not. The question is whether one compromised model decision can cross a hard system boundary.

Enterprise teams need to design on the assumption that an injection will eventually reach the model. The right controls make that event observable, bounded, and recoverable.

Prompt Injection Turns Business Content Into a Control Channel

A normal application separates code from data. A database value does not become a SQL command unless the application handles it unsafely. Language models do not provide the same clean boundary. System instructions, a user's request, retrieved text, and a tool response can all arrive as tokens in one context.

The model may be told that system instructions have higher authority, but this is a learned behavior rather than an access-control mechanism. Hostile text can impersonate a policy update, claim that an emergency exception applies, or hide an instruction inside content the user genuinely asked the agent to inspect.

Direct injection comes from the person talking to the model. Indirect injection comes through another source:

• An email footer tells the assistant to forward the conversation.
• A support ticket asks the agent to reveal its system prompt and customer data.
• A webpage contains hidden text that changes a research agent's objective.
• A PDF instructs an invoice agent to replace payment details.
• A calendar invitation asks the assistant to invite an external attendee to private meetings.
• A tool result contains an instruction that looks like the next step in the workflow.

The AgentDojo benchmark made this concrete with 97 realistic tasks and 629 security test cases across email, banking, travel, and other tool-using scenarios. Its useful finding for a CISO is not a leaderboard position. It is that useful task completion and security have to be measured together. A defense that blocks every tool call is secure but useless. An agent that completes every task while obeying injected commands is useful only to the attacker.

NIST's 2025 adversarial machine learning taxonomy recommends planning around the possibility of prompt injection when models interact with untrusted sources. That assumption changes the architecture. Retrieved content should inform an action, never grant authority to perform it.

Consider three statements in the same agent context:

The model may reason over all three. The surrounding application must enforce the authority distinction.

Permissions Are Necessary, but They Are Not the Whole Defense

Many security reviews stop at role-based access control: the agent can only do what its service account permits. That is a good start and a dangerous stopping point. A perfectly permissioned agent can still misuse every permission it legitimately has.

If an email assistant is allowed to read every message and send email, an injection only needs to persuade it to combine those two valid capabilities. If a finance agent can read vendor records and update bank details, the attack does not require privilege escalation. The available privilege is already enough.

Build the control stack in layers:

The OWASP prompt-injection guidance recommends least privilege, external-content segregation, output validation, human approval, and adversarial testing. These controls reinforce one another. None should depend on the model remembering a warning from its prompt.

Tool design matters more than tool descriptions. Avoid a generic execute_action(payload) function with broad credentials. Give the agent narrow operations such as get_invoice, draft_vendor_email, and request_payment_change. Put authorization and value limits inside those functions. A model can propose an argument; trusted code decides whether the argument is legal. The companion playbook on agent guardrails and policy enforcement shows how to decide which rules belong at each enforcement point.

Run the action in the user's context whenever possible. CIS Control 6 calls for access rights to be created, assigned, managed, and revoked according to enterprise policy. That includes the credentials an agent uses on a person's behalf. A request from an analyst should not inherit the controller's rights because both use the same assistant.

Our guide to permission-aware AI agents covers retrieval filtering and identity propagation in more detail. Prompt-injection controls sit on top of that foundation. They do not replace it.

Use a Four-Gate Action Pipeline

A production agent should not move directly from model output to side effect. Put each proposed action through four gates: provenance, authorization, validation, and confirmation.

Gate 1: Provenance. Record what caused the proposal. Which user asked? Which documents were retrieved? Which tool results were shown? Which policy version applied? Provenance does not prove that the action is safe, but it makes suspicious influence visible and gives reviewers evidence.

Gate 2: Authorization. Check whether the authenticated user can perform this exact action against this exact resource. Do not ask the model to infer access. Enforce authorization in the destination system or a policy service. Deny attempts to expand scope based on retrieved content.

Gate 3: Validation. Validate schema, target, value, volume, destination, and business rules. A payment tool should reject an unknown bank account. An email tool should block new external domains unless policy allows them. A CRM tool should cap bulk updates and require an idempotency key. These checks are ordinary software and should behave deterministically.

Gate 4: Confirmation. Ask for human approval when the remaining risk exceeds a defined threshold. The approval screen should say what will happen, not merely ask whether the agent may "continue." Show the recipient, amount, records affected, source evidence, and rollback options.

The resulting contract can be written as a decision table:

This structure also protects against ordinary hallucination and ambiguous user requests. Prompt injection is one cause of unsafe model output, not the only cause. Minimize tools, functionality, permissions, and autonomy, then require approval for high-impact work.

A useful implementation detail is to split planning from execution. The planner can read broadly and produce a typed proposal without write credentials. A separate executor receives only the validated action, not the full untrusted context. For highly sensitive workflows, use another deterministic policy check or narrowly scoped model to compare the proposal with the authenticated request before any side effect occurs.

Test the Attack Path, Not Just the Model

Prompt-injection testing often degenerates into a list of clever phrases. That misses the enterprise problem. The attack surface is the complete route from content ingestion to business action.

Start with a content and capability inventory:

1. List every source the agent can read: email, chat, PDFs, webpages, tickets, code, images, OCR, calendar events, CRM notes, and tool responses.
2. List every action it can take and the credential used for each action.
3. Mark which sources an external party, customer, vendor, or lower-privileged employee can influence.
4. Connect each source to the actions it could affect in the current workflow.
5. Rank paths by data sensitivity, reversibility, financial impact, and blast radius.

Then build attacks against the paths that matter. Put instructions in visible text, metadata, quoted email threads, HTML, white-on-white PDF text, image text, and results returned by a mock tool. Try instructions that request a new objective, claim special authority, split the payload across sources, or tell the agent to hide its action from the user.

Score four outcomes for every case:

• Utility: Did the agent still complete the legitimate user task?
• Security: Did it reject the attacker's objective?
• Confidentiality: Did sensitive data remain inside the permitted boundary?
• Action safety: Were unauthorized tool calls blocked before execution?

Do not grade only the final prose. Inspect tool calls, denied calls, arguments, approvals, and downstream state. Our production evals guide explains how to turn traces into regression cases and release gates.

Red-team after changing the model, system prompt, retrieval pipeline, tools, memory, or policy. A stronger model is not automatically safer. A large 2025 public competition recorded 1.8 million injection attempts and more than 60,000 policy violations, with the resulting NeurIPS security study reporting little correlation between robustness and model capability, size, or inference compute.

Release only when legitimate-task performance and security performance both clear their thresholds. Track attempted injections, blocked actions, approval overrides, false positives, sensitive-data exposures, and time to investigate. A security control that operators routinely bypass needs redesign, even if its benchmark score looks good.

An Enterprise Rollout That Limits the Blast Radius

Start with a workflow where read and write boundaries are easy to see. Invoice triage is reasonable. Autonomous bank-detail changes are not. Support research is reasonable. Unreviewed customer refunds across every account are not.

Use three rollout stages.

Shadow mode: The agent reads real inputs and proposes actions, but cannot execute. Compare proposals with operator decisions. Seed adversarial content and verify that the agent's plans, logs, and alerts reveal it.

Draft mode: The agent may prepare messages, updates, or transactions in a staging area. A human commits them through the destination system. Measure how often reviewers materially change the draft and whether the approval view exposes the right evidence.

Bounded action: Allow a narrow set of reversible, low-value actions. Put hard limits around tool, tenant, record count, amount, recipient, and time window. Expand one boundary at a time only after the eval set, incident process, and rollback mechanism cover it.

Assign ownership before launch. Security owns the threat model and incident standard. The business owner defines harmful outcomes and approval thresholds. Engineering owns policy enforcement, logging, and rollback. Operators own acceptance tests and report surprising behavior. The model provider owns none of those company-specific decisions.

An incident runbook should include credential revocation, tool disablement, affected-record identification, evidence preservation, downstream rollback, user notification criteria, and conversion of the attack into a permanent test. If disabling the agent requires a deployment, the kill switch is too slow.

How OpenNash Can Help

OpenNash helps enterprise teams turn an agent-security review into controls that are wired into the workflow. We map untrusted inputs, identities, tools, sensitive data, and consequential actions. Then we design the trust boundaries, scoped credentials, policy checks, approval gates, traces, and rollback paths that contain a compromised decision.

The work includes an adversarial test set built from the documents and messages the agent actually sees. We test task completion and attack resistance together, connect failures to release gates, and leave the team with code, policies, eval cases, and an incident runbook it can own.

An existing platform may already provide the identity, approval, and audit controls this workflow needs. Custom enforcement becomes necessary when permissions are unusually fine-grained, systems are proprietary, or write actions carry high consequences. In either case, do not grant autonomy until the team can name the user, source, allowed action, and rollback path.

Book a call to map this to one workflow. Bring the tools the agent can call, the content it reads, and the three actions you would most regret seeing it take incorrectly.