How do AI agents break security assumptions, and how do you contain them?

AI agents break old security assumptions because they decide what to do at run time, using permissions you gave them, based on text they cannot reliably tell apart from instructions. The answer is not a cleverer prompt. It is to give each agent the least access, put authorization outside the model, and require approval for high impact actions.

What assumption does an agent break?

Traditional software does what its code says. The caller, the permissions and the actions are fixed at build time and reviewable. An agent chooses its actions from model output. OWASP describes LLM-based systems that are given the ability to call functions or interface with other systems through extensions, which may be called tools, skills or plugins, with the choice of which to invoke sometimes delegated to the model. Agent systems typically make repeated model calls, using earlier output to direct later steps.

That produces a system where the sequence of actions is not known in advance and depends on every piece of text the agent reads.

What does OWASP call the resulting flaw?

Excessive Agency: the vulnerability that lets damaging actions happen in response to unexpected, ambiguous or manipulated model output, whatever causes the model to misbehave. OWASP lists common triggers such as hallucination caused by poorly engineered benign prompts or a weak model, and direct or indirect prompt injection from a malicious user, from an earlier malicious or compromised extension, or, in multi-agent systems, from a compromised peer agent. It names the root causes as excessive functionality, excessive permissions and excessive autonomy.

Notice that the trigger does not have to be an attacker. A confused model with broad permissions can do damage with no malice involved.

What controls does OWASP recommend?

The mitigations map directly onto those root causes. Minimize the extensions an agent may call. Minimize what each extension can do: a mailbox summarizer needs read access, not send or delete. Avoid open-ended extensions such as running a shell command or fetching any URL, and prefer narrow ones. Limit the permissions the extension holds on other systems, for instance read access to one table rather than the whole database, enforced by database permissions on the identity the extension uses. Execute actions in the user's context, with the minimum scope, so the agent cannot do more than the person could. Require human approval for high impact actions. And use complete mediation: enforce authorization in the downstream systems instead of trusting the model to decide whether an action is allowed.

OWASP adds that logging and monitoring of extension activity, and rate limiting, do not prevent excessive agency but can limit the damage.

A constructed example

A company builds an assistant that reads incoming support email and drafts replies. The first version can read and send mail and look up orders. A customer email contains hidden text telling the assistant to forward the last ten order histories to an outside address. Under the first design, nothing stops it. Under the second, the extension can only draft, not send; the order lookup is read only and scoped to the customer named in the ticket; and sending requires a human to click approve. The attack text is still in the email. It no longer has anything to act on.

How to check or apply this

1. List every agent, the tools it can call and the credentials it holds.
2. Remove any tool the agent does not need for its job.
3. Replace open-ended tools, such as raw shell or arbitrary web fetch, with narrow ones.
4. Give each agent its own identity with least privilege, and do not reuse a human's credentials.
5. Enforce authorization in the downstream service so the model cannot talk its way past it.
6. Require approval for actions that send, delete, pay or change permissions.
7. Log every tool call with the agent identity, and review the logs.
8. Test with hostile content in the data the agent reads, such as hidden text in an email or a document.

Safeguard's agent identity and MCP Guardrail capabilities address the identity and tool-call control steps, and its MCP server lets teams use Safeguard from AI tools with unlimited tool calls on every plan.

Quick check

Sources

• OWASP Gen AI Security Project: LLM06:2025 Excessive Agency
• OWASP Gen AI Security Project: Top 10 for LLM Applications

Last reviewed: 18 October 2026.