Scope an AI red team exercise in three phases: plan and scope, execute, then assess, report and improve. MITRE ATLAS describes this structure in its AI Red Team mitigation. The scoping phase does most of the work, because it fixes what you test, what is off limits and how you will know the result.
Phase 1: plan and scope
ATLAS lists what to write down before any testing starts.
The system. Intended use, deployment environment, users, sensitive data, connected resources and what happens if it fails or is misused. Include a diagram of components, trust boundaries, data flows, external services and human decision points.
Rules of engagement. Which systems, accounts, data and techniques are authorised, the test windows, resource limits, escalation steps, evidence handling and stop conditions. ATLAS says to plan destructive, privacy-invasive or high-cost tests for isolated environments with safeguards.
A threat model. Who the adversary is, what they want, what access and knowledge they have, and what constraints apply. ATLAS says to use its tactics, techniques and procedures to build the threat vectors and rank them by likelihood and severity.
Success criteria. Task-level metrics for how the AI capability is affected, and operational metrics for the wider system. Decide this now. Otherwise results get argued about afterwards.
Cover the whole system
The ATLAS text is clear that the exercise should consider more than the model: data, agents including memory and tools, data flows, decision processes, application logic, retrieval systems, identities and permissions, software dependencies, infrastructure, user interfaces and human workflows.
A RAG assistant test that only sends clever prompts to the chat box covers a small slice. Include the document sources, the ingestion job and the permissions on the vector store.
Phase 2: execute
Use both manual and automated methods. ATLAS notes that automation can generate input variations, replay attack sequences and evaluate responses at scale, while human testers build system-specific attacks and adapt to the defences they see.
During the run:
- Follow the rules of engagement and record attack activity, system responses and control behaviour.
- Stop or escalate when a pre-defined condition is hit.
- Afterwards, remove test accounts, modified data, installed software and persistent instructions.
That last point is easy to forget with AI systems. A poisoned test document left in an index is a live attack on your next user.
Phase 3: assess, report, improve
Compare results with the criteria you set. Record successful and failed attacks, their consequences, how the controls behaved and gaps in the threat model. Report to developers, defenders, operations and risk owners. Give each finding an owner, track the fix and retest the corrected system.
ATLAS also says to use demonstrated attacks to improve the next round of preparation. The exercise is meant to recur, not to be a one-off audit.
A short scoping template
- System and diagram, one page.
- Authorised and forbidden actions, one page.
- Top five threat vectors with a likelihood and impact note each.
- Metrics and stop conditions.
- Cleanup list.
- Names of the people who own fixes.
If you cannot fill in item 6, the exercise will produce a report and nothing else.
Quick check
How often should an AI red team exercise run?
ATLAS describes recurring exercises, before deployment and throughout operation. A sensible trigger is any large change to data sources, tools or the model.
Can automation replace human testers?
ATLAS describes them as complementary: automation for scale, humans for system-specific attacks and unexpected behaviour.
Sources
- MITRE ATLAS data repository (mitre-atlas/atlas-data), mitigation AML.M0035 AI Red Team, 2026.09 release.
Last reviewed: 19 October 2026.
