Safeguard
AI Security

Guardrails for Autonomous AI Agents: Allowlisting, Validation, and Human-in-the-Loop

OWASP's 2025 LLM Top 10 splits Excessive Agency into three root causes. Here's how tool allowlisting, output validation, and approval gates address each one.

Safeguard Research Team
Research
6 min read

In 2025, the OWASP Gen AI Security Project revised its Top 10 for LLM Applications and significantly expanded one entry: LLM06:2025, Excessive Agency. The update didn't just bump a severity score — it broke a vague "agents can do too much" warning into three distinct, addressable root causes: excessive functionality (an agent can reach tools it has no task-related reason to touch), excessive permissions (the tools it calls run with broader privileges than the task needs), and excessive autonomy (high-impact actions execute without any human checkpoint). OWASP's writeup cites the pattern concretely: an agent wired up with email access gets manipulated by a crafted input into sending phishing messages, or one with database access gets talked into deleting records it was only supposed to read. Neither failure requires a novel exploit — both come from an agent that technically works as designed, just with more reach than the task warranted. As autonomous agents move from demos into production pipelines that touch ticketing systems, codebases, and customer data, that three-way breakdown is a useful map for where guardrails actually need to sit. This post walks through it: what each root cause looks like in practice, and the concrete controls — tool allowlisting, output/permission validation, and human-in-the-loop checkpoints — that constrain it.

What does "excessive functionality" look like in a real agent deployment?

Excessive functionality shows up whenever an agent is handed a general-purpose toolset for a narrow job. A support-ticket triage agent that gets the same MCP server connection as a full ops agent — with tools for refunds, account deletion, and infrastructure changes — has functionality the triage task never calls for, and every one of those unused tools is attack surface if a crafted ticket ever manages to redirect the agent's behavior. OWASP's framing is explicit that this is a scoping failure, not a model failure: the fix isn't a smarter model, it's giving the agent a smaller toolbox. The practical control is a named allowlist — deciding, per agent or per task, exactly which tools it can invoke, and denying everything else by default rather than trusting the agent's own judgment to skip tools it "shouldn't" need. That decision has to be made and enforced by infrastructure the agent doesn't control, because an agent that can talk itself into calling a tool is also an agent that could be talked into it by someone else's input.

How does excessive permission differ from excessive functionality, and why does it matter separately?

Excessive functionality is about which tools an agent can see; excessive permission is about how much damage the tools it's already allowed to use can do. An agent legitimately needs a "delete_ticket" tool for its job, but if that tool runs under a service credential that can delete any record in the system rather than just tickets the agent created, the blast radius of a single manipulated call is far larger than the task requires. OWASP separates these because the fixes are different: allowlisting narrows which tools are reachable, while permission scoping narrows what each reachable tool is capable of — least-privilege service accounts, row-level scoping, read-only credentials wherever write access isn't the actual job. A tool call that returns unexpected data (a customer record from outside the requester's scope, a file path outside an expected directory) is also a signal worth validating on the output side before the agent acts on it further, not just trusting because the call itself was "allowed."

How does the Model Context Protocol build human-in-the-loop into its design?

Anthropic's Model Context Protocol (MCP) — the emerging standard for connecting LLM applications to external tools and data — doesn't treat human oversight as an optional add-on; it names consent as a core security principle for every implementation. Per the MCP spec published at modelcontextprotocol.io, tools are model-invoked, but the spec states that hosts must obtain explicit user consent before invoking any tool on the model's behalf — and while MCP acknowledges it cannot mechanically enforce that at the protocol level, it holds host applications to the same explicit-approval standard for sampling requests (cases where a server asks the connected LLM to generate something). Claude Desktop's reference implementation puts this into practice with per-server approval prompts and confirmation before sensitive operations. Claude Code takes a complementary, narrower approach at the developer-configuration layer: rather than relying solely on a runtime prompt, it requires developers to name which MCP tools are permitted at all, ahead of time. Together these show the same principle applied at two different points — consent at the moment of the call, and allowlisting before the call is even possible.

Where should a human-in-the-loop checkpoint actually sit in an agent workflow?

The OWASP framing on excessive autonomy is specifically about high-impact or irreversible actions proceeding without anyone reviewing them first — which means the checkpoint has to sit before the irreversible step, not somewhere generic like "at the start of the session." A checkpoint that only asks a human to approve the agent's overall goal ("yes, go handle this ticket") doesn't cover a later step where the agent decides, three tool calls in, to issue a refund or delete an account. The MCP consent model gets this right by scoping approval to the individual tool invocation, not the session. In practice, that means classifying tools by impact ahead of time — read-only lookups can run unattended, while anything that sends, deletes, pays, or modifies production data waits for an explicit approval — and building the approval gate into the tool-calling layer itself rather than trusting a system prompt instruction to the model to "ask before doing anything risky."

How Safeguard Helps

Guard, Safeguard's runtime layer for MCP traffic, gives teams the inventory these guardrail patterns depend on: every MCP server and every agent is a registered record, not an assumed identity, so policy can target a specific server or a specific agent instead of applying blindly across the org. Each agent carries a running risk score and anomaly score that Guard maintains from observed behavior, and when an agent's calls look like a functionality or permission overreach, it can be blocked, unblocked, or quarantined through a dedicated lifecycle action — a direct, auditable containment step rather than a policy change that takes a deploy cycle to land. Griffin AI Search's own autonomous browser agent applies the same philosophy to itself: a hard cap of 30 actions or 60 seconds per session, and a same-origin check that defaults to suggestion-only — drawing an overlay instead of dispatching a click — whenever it can't confirm it's safe to act directly. That fail-safe-to-suggestion default is the human-in-the-loop pattern this post describes, built into the product rather than left to a prompt.

Never miss an update

Weekly insights on software supply chain security, delivered to your inbox.