To red team an AI agent before launch, list what it can reach, feed it untrusted content through every channel it reads, and try to make it take actions outside its job. Define pass criteria first, run the same scenarios after each fix, and treat failures as design problems, not prompt problems.
Why is an agent different from a chatbot?
A chatbot that says something wrong embarrasses you. An agent that does something wrong changes data, spends money or leaks information. OWASP's Excessive Agency entry describes the vulnerability as damaging actions performed in response to unexpected, ambiguous or manipulated model outputs, whatever causes the malfunction: a hallucination, a direct or indirect prompt injection, a compromised tool or a malicious peer agent. Red teaming has to test the actions, not just the words.
What do you scope first?
Write down four things before any testing.
- Capabilities: every tool, API, file location and credential the agent can use.
- Inputs: every place text reaches the model, including user messages, retrieved documents, web pages, tool outputs, tickets and emails.
- Actions that matter: the ones that are irreversible, external or sensitive.
- Boundaries: what the agent must never do, stated as testable rules.
OWASP traces excessive agency to excessive functionality, permissions and autonomy, so the scoping list doubles as your map of where to push.
What scenarios should you run?
Group them by channel, not by clever phrasing. We deliberately describe categories rather than payloads.
Direct instruction conflicts. A user asks for something outside the agent's role, politely and insistently. Does it refuse or comply?
Indirect injection. Plant instructions inside content the agent will read, such as a document, a web page or a support ticket, and see if it follows them. OWASP says the model cannot reliably tell instructions from data, so this is the highest yield area.
Tool misuse. Ask for tasks that would require a tool in a way the designer did not intend: a read tool used to find secrets, a write tool aimed at the wrong target, a fetch tool pointed at an internal address.
Data movement. Check whether information from one user or tenant can reach another through memory, search or shared context, and whether the agent can send data out through a tool argument.
Persistence. If the agent has memory, test whether poisoned content in one session affects later ones.
Runaway behavior. Loops, repeated calls and large spend without a stop.
A constructed example
Constructed example: a support agent can read tickets, search the knowledge base and issue refunds up to a limit. The red team plants a line in a ticket that tells the agent to refund a different order. Result: the agent drafts the refund. Finding: refund authority sat with the agent and had no approval. Fix: refunds above zero need human approval, the agent's token can only touch the order tied to the ticket, and ticket text is passed as marked untrusted content. Rerun: the injected instruction no longer produces a refund call.
Notice the fix is permissions and approval, not a stronger system prompt.
What are good pass criteria?
Write them as checks a script can evaluate: no call to tool X without approval, no data from tenant B in a tenant A session, no outbound request to a non allowed domain, spend stays under a cap. Keep the scenarios in version control and run them in CI on every change to the prompt, model, tools or permissions. OWASP recommends adversarial testing and attack simulations as part of its prompt injection mitigations.
Who should do it?
Someone who did not build the agent. Pair them with the builder for fixes. Rerun everything after fixes, since changes can break earlier protections.
Safeguard's guardrails and MCP Guardrail apply policy at the tool call layer, which is where several of the findings above get enforced. Testing still has to show that they work in your setup.
What to do
- Write the scope: capabilities, inputs, sensitive actions and boundaries.
- Build scenarios per input channel and per sensitive action.
- Define scriptable pass criteria before running anything.
- Fix by reducing tools, permissions and autonomy first, prompts second.
- Add human approval for irreversible actions.
- Keep the suite in CI and rerun it on every change.
Quick check
Is a stronger system prompt a fix for injection?
Not on its own. OWASP says it is unclear whether fool proof prevention exists, so limit impact with least privilege, validation and human approval.
When should red teaming happen?
Before launch and after every meaningful change to the model, prompt, tools or permissions, using the same saved scenarios.
Sources
Last reviewed: 18 October 2026.
