How to Threat Model an AI Agent: A Step-by-Step Guide

To threat model an AI agent, draw its data flows, list every tool and permission it holds, mark which inputs are untrusted, enumerate what could go wrong at each step, assign controls such as least privilege and human approval, and retest after changes. The agent's actions, not its words, are the main risk.

Key takeaways

• Treat the model as an untrusted component that can be steered by any text it reads.
• Excessive agency is the center of agent risk: too many tools, too many permissions, too much autonomy.
• Microsoft's AI/ML threat modeling guidance adds training data, model supply chain and query access to the scope.
• Enforce authorization in downstream systems, not in the prompt.

What makes an AI agent different to threat model?

An agent reads untrusted text and then acts. OWASP LLM06:2025 defines excessive agency as damaging actions taken because outputs were unexpected, ambiguous or manipulated, with triggers such as hallucination and prompt injection from users, compromised extensions or peer agents.

OWASP's Agentic Security Initiative published a threat-model-based reference of emerging agentic threats and mitigations on 17 February 2025. Use it alongside this guide.

Step 1: What are you protecting?

List assets: customer data, credentials, code, money movement, reputation. Note what the agent can reach, since that is the blast radius.

Step 2: How does data flow?

Draw a data flow diagram with these boxes: user, agent loop, model, tools, memory, retrieval store, external content, outputs. Mark trust boundaries wherever content crosses from a source you do not control, such as web pages, emails, tickets or documents.

Step 3: What tools and permissions does it hold?

Make a table of each tool, its credentials and its worst case action.

OWASP names the root causes: excessive functionality, excessive permissions and excessive autonomy.

Step 4: What can go wrong?

For each flow and tool, ask these questions, drawn from Microsoft's guidance and OWASP:

• If the data or knowledge base is poisoned, how would you know?
• Can a document or message instruct the agent to call a tool?
• Can the model output sensitive data, or reveal how it works?
• Which third party models, data providers and packages sit in the supply chain?
• What happens if someone floods the agent with requests?

Microsoft's guidance, dated November 2019, lists threats such as data poisoning, model inversion, model stealing, backdoored models and attacks on the ML supply chain. It also says traditional security threats remain essential, because failure to address them helps enable the AI-specific attacks.

Step 5: What controls reduce each risk?

OWASP's prevention advice for excessive agency gives the starting list:

1. Offer only the tools the task needs.
2. Prefer narrow tools over shell access or arbitrary URL fetching.
3. Use least-privilege credentials, scoped to the individual user.
4. Require human approval for sending, posting or deleting.
5. Enforce authorization in downstream systems, not in the model.
6. Log tool activity and rate limit.

The MCP guardrail demo shows a check placed between an agent and its tools.

Click through it at your own pace, or open it full screen.

Step 6: How do you test the model?

Turn each threat into a test. Use a red teaming tool such as garak or DeepTeam against staging, and add scripted checks that an injected document cannot trigger a tool call. See LLM red teaming tools compared. Repeat when tools, prompts or permissions change.

Frequently asked questions

Sources

• OWASP LLM06:2025 Excessive Agency
• OWASP Agentic AI Threats and Mitigations
• Microsoft: Threat Modeling AI/ML Systems and Dependencies
• OWASP Top 10 for LLM Applications

Last reviewed: 19 October 2026.