AI agents and systemic risk: ten failure modes to plan for

AI agents fail in ten recurring ways: hijacked instructions, excess permissions, unchecked output, poisoned retrieval, compromised tools, confused identity, runaway consumption, overreliance, shared dependencies and missing oversight. Most have known controls. The systemic part is that one weak point can spread across many agents that share the same model, tools or credentials.

This is a calm explainer. It predicts nothing and names no incident. Each item points to published guidance from OWASP, the Model Context Protocol project or NIST.

What does systemic risk mean here?

In finance, systemic risk is a failure that spreads because parts of the system depend on each other. The same idea applies to agents. If twenty agents in a company share one model provider, one tool server and one service account, a single problem can show up in all of them at once. The failure modes below are ordinary. Sharing is what makes them larger.

The ten failure modes

1. Hijacked instructions. OWASP LLM01 describes prompt injection, including indirect injection, where content from a website or file changes the model's behavior. OWASP says it is unclear whether fool-proof prevention exists, so the aim is to limit impact. Control: treat external content as untrusted and keep it separate from instructions.
2. Excess permissions. OWASP LLM06 names three root causes of excessive agency: excessive functionality, excessive permissions and excessive autonomy. Control: give each agent the fewest tools and the narrowest credentials that the task needs.
3. Unchecked output. OWASP LLM05 covers model output passed to other systems without validation, for example into a shell, a database query or a browser. Control: treat output like input from any user and validate or encode it for where it goes.
4. Poisoned or leaky retrieval. OWASP LLM08 covers weaknesses in how embeddings and retrieved documents are stored and fetched, including cross-tenant leaks and poisoned sources. Control: permission-aware stores, vetted sources and retrieval logs.
5. Compromised tools and components. OWASP LLM03 covers supply chain risks across models, datasets and platforms. The MCP security guidance describes local servers that run commands with the client's privileges. Control: vet suppliers, keep an inventory, and review commands before approving a local tool server.
6. Confused identity. The MCP guidance describes confused deputy problems and token passthrough, where a server forwards a credential that was not issued to it. Control: validate token audience and obtain separate credentials for downstream calls.
7. Runaway consumption. OWASP LLM10 covers unbounded consumption, including a "denial of wallet" pattern that exploits per-use pricing. Control: rate limits, quotas, timeouts and spend alerts per agent.
8. Overreliance. OWASP LLM09 notes that users who trust output without checking it make misinformation worse. Control: label AI output, define what a human must verify, and measure how often reviewers disagree with the agent.
9. Shared dependencies. This one is our inference from the other items. Many agents on one provider, one tool server or one service account share a single point of failure. NIST AI RMF GOVERN 6.2 asks for contingency processes for failures in third-party data or AI systems deemed high-risk. Control: write down what each agent depends on, and what happens when that dependency fails or changes.
10. Missing oversight. NIST AI RMF MANAGE 4.1 asks for post-deployment monitoring plans covering user input, appeal and override, incident response and change management, and the MCP tools guidance says clients should log tool usage for audit. Control: log what agents do, set alerts, and keep a way to pause one agent or all of them.

How do the failures chain together?

A realistic chain, described in general terms, runs like this. An agent reads a web page that contains instructions (mode 1). The agent holds a broad credential (mode 2). It passes a generated command to a system without checking it (mode 3). Nobody notices because logging is thin (mode 10). Each step has a known control, and any one of them breaking the chain is enough.

That is the argument for layers. No single control is reliable on its own, so the aim is to make sure that failing one layer does not give an attacker or a mistake everything.

What should a calm response look like?

Do not freeze projects. Do not pretend the risks are solved. Treat agents like a new class of employee with powerful tools: limited access at first, review of their work, and growing trust as evidence builds.

What to do

1. Make a one page list of every agent in use: purpose, tools, credentials, model provider, owner.
2. For each, write the worst thing it could do with its current permissions. Reduce permissions until that sentence is acceptable.
3. Add confirmation for actions that send data out, delete, spend money or change access.
4. Turn on logging for tool calls and set alerts on unusual volume.
5. Set spend and rate limits per agent.
6. Write the fallback for each shared dependency, and test the pause switch once.
7. Review the list every quarter, and after any incident.

Quick check

Sources

• OWASP LLM01:2025 Prompt Injection
• OWASP LLM03:2025 Supply Chain
• OWASP LLM05:2025 Improper Output Handling
• OWASP LLM06:2025 Excessive Agency
• OWASP LLM08:2025 Vector and Embedding Weaknesses
• OWASP LLM09:2025 Misinformation
• OWASP LLM10:2025 Unbounded Consumption
• MCP specification: Security Best Practices
• MCP specification: Tools
• NIST AI 100-1, AI RMF 1.0

Last reviewed: 18 October 2026.