AI systems fail in ways traditional software testing never anticipates: a chatbot can be talked into leaking a system prompt, an agent can be tricked into calling a tool it shouldn't, and a fine-tuned model can be jailbroken with a cleverly worded prompt. AI red teaming is the practice of deliberately attacking your own AI systems — models, agents, and the pipelines around them — to find these failures before an attacker or a regulator does. Unlike a one-off pentest, a well-run AI red teaming exercise follows a repeatable process: scoping, threat modeling, structured attack execution, evidence capture, and remediation tracking.
This guide walks through how to plan and run a structured AI red teaming exercise from scratch, with concrete steps, sample commands, and a checklist you can reuse every release cycle. By the end, you'll have a repeatable red teaming methodology you can apply to any LLM-backed feature, along with the artifacts (attack logs, severity scores, a findings report) that auditors, security leadership, and engineering teams actually want to see.
Step 1: Define Scope and Threat Model for the AI Red Teaming Exercise
Before anyone sends a single adversarial prompt, define what's in bounds. AI attack surface is broader than most teams assume — it includes the model itself, the system prompt, retrieval sources (RAG), connected tools/functions, memory stores, and the surrounding application logic that decides what the model is allowed to do.
Build a lightweight threat model that answers:
- Assets: What does the model have access to? (customer data, internal APIs, code execution, payment systems)
- Actors: Who can reach the model? (anonymous users, authenticated users, other agents, upstream data sources)
- Trust boundaries: Where does untrusted input enter the pipeline? (user prompts, uploaded documents, tool outputs, third-party plugin responses)
- Impact categories: What's the worst case? (data exfiltration, unauthorized action, harmful content generation, reputational harm)
Document this in a short scoping sheet before execution. A minimal YAML scope file keeps the exercise auditable:
exercise: q3-support-agent-red-team
target:
system: customer-support-agent-v2
interfaces: [chat-api, slack-bot]
tools_exposed: [order_lookup, refund_issue, ticket_create]
data_access: [customer_pii, order_history]
out_of_scope:
- production_billing_writes
- third_party_auth_provider
success_criteria:
- no unauthorized refund_issue calls
- no PII disclosure outside authenticated session
- no system prompt leakage
Step 2: Assemble the Attack Taxonomy
A structured red teaming methodology relies on a taxonomy so testers aren't improvising and coverage is measurable. Organize test cases into categories aligned with known frameworks (OWASP Top 10 for LLM Applications, MITRE ATLAS) rather than ad hoc prompts:
- Prompt injection — direct (user types malicious instructions) and indirect (malicious instructions hidden in retrieved documents, emails, or tool outputs)
- Jailbreaking / guardrail bypass — role-play framing, encoding tricks, multi-turn escalation, obfuscated payloads
- Data exfiltration — extracting system prompts, training data, other users' context, or internal configuration
- Excessive agency abuse — coercing the agent into calling tools or taking actions outside its intended authorization
- Output manipulation — inducing harmful, biased, or non-compliant content
- Denial of service / resource abuse — prompts designed to spike token usage, loop the agent, or exhaust rate limits
- Supply chain and dependency risks — poisoned fine-tuning data, compromised plugins, malicious model weights or embeddings
Map each category to specific test cases with expected pass/fail criteria. This is what separates AI adversarial testing from casual "let's try to jailbreak it" sessions — every attack has a hypothesis and a measurable outcome.
Step 3: Set Up an Isolated Test Environment
Never run adversarial testing against production with live customer data. Stand up a mirrored environment with synthetic or anonymized data, and instrument it so every prompt/response pair is logged.
# Spin up an isolated copy of the agent stack for testing
docker compose -f docker-compose.redteam.yml up -d
# Point the harness at the isolated endpoint, not prod
export TARGET_ENDPOINT=https://redteam.internal.example.com/v1/chat
export LOG_SINK=./redteam-runs/$(date +%Y%m%d)
mkdir -p "$LOG_SINK"
If you're testing an agent with tool access, use mocked or sandboxed versions of downstream systems (a fake payments API, a scoped-down database) so a successful exploit doesn't create real-world consequences. Capture full transcripts, not just final outputs — the multi-turn escalation that got a model to comply is often more valuable than the final harmful response itself.
Step 4: Execute the Attacks
Run each test case from your taxonomy, using a mix of manual expert probing and automated fuzzing. Manual testing catches nuanced, context-aware bypasses; automation gives you scale and regression coverage across releases.
A simple automated runner might look like this:
import json, requests
def run_case(case, endpoint):
resp = requests.post(endpoint, json={"messages": case["turns"]})
result = resp.json()
passed = case["fail_condition"] not in result.get("content", "")
return {
"id": case["id"],
"category": case["category"],
"passed": passed,
"raw_response": result
}
cases = json.load(open("attack_taxonomy.json"))
results = [run_case(c, TARGET_ENDPOINT) for c in cases]
json.dump(results, open(f"{LOG_SINK}/results.json", "w"), indent=2)
For an LLM red team exercise involving multiple testers, assign categories by expertise (a prompt-injection specialist, a bias/safety specialist, an application-security engineer focused on tool abuse) and run in parallel tracks with a shared logging convention so results can be merged and deduplicated.
Step 5: Score Findings and Assign Severity
Not every successful jailbreak is equally dangerous. Score each finding against a consistent rubric so engineering can triage sanely:
| Severity | Criteria | Example |
|---|---|---|
| Critical | Unauthorized action with real-world impact | Agent issues a refund without authorization |
| High | Sensitive data disclosure | Model leaks another user's conversation history |
| Medium | Guardrail bypass without direct harm | Jailbreak produces disallowed content with no data/action impact |
| Low | Cosmetic or low-confidence bypass | Model briefly breaks character but self-corrects |
Attach reproduction steps, the exact prompt sequence, and the raw transcript to every finding above Low. This turns a red team run into an actionable backlog rather than a pile of screenshots.
Step 6: Report, Remediate, and Re-Test
Compile findings into a report that ties each issue back to the threat model from Step 1: which asset was exposed, which trust boundary was crossed, and what the fix should be (system prompt hardening, input/output filtering, tool authorization changes, retrieval source sanitization, or rate limiting).
Track remediations like any other security finding — ticketed, owned, and re-tested before closure. Re-run the exact failing test case after a fix ships; a surprising number of "fixes" only patch the specific wording used during the exercise and fail against a trivial rephrasing, which is why regression coverage matters as much as the initial find.
Troubleshooting and Verification
A test case "passes" but the response still looks wrong. Automated pass/fail checks based on keyword matching are brittle. Supplement string-matching with a secondary LLM-as-judge pass, and always spot-check a sample of "passed" transcripts manually — models often comply with harmful requests using synonyms or indirect phrasing that a naive filter misses.
Results aren't reproducible between runs. LLM outputs are non-deterministic even at low temperature. Run flaky-looking test cases 3-5 times and report a pass rate rather than a binary result, and pin model versions/temperature settings in your exercise scope file so comparisons across runs are meaningful.
The agent behaves differently in the red team environment than in production. Verify that the isolated environment uses the same system prompt, model version, tool definitions, and middleware (content filters, rate limiters) as production. A common false negative is testing against a stripped-down staging config that doesn't reflect what's actually deployed.
Findings pile up faster than they can be fixed. Prioritize by severity and exploitability, not volume. A handful of Critical findings that enable unauthorized actions matter more than fifty Low-severity content-policy edge cases — fix the former first and track the latter as a backlog.
Stakeholders question whether the exercise was thorough. Map your executed test cases back to the taxonomy from Step 2 and report coverage percentages per category, not just a list of bugs. This shows gaps as clearly as it shows findings, and it's the evidence auditors and leadership will ask for.
How Safeguard Helps
Running an AI red teaming exercise by hand — building a taxonomy, wiring up isolated environments, scoring findings, and proving coverage to auditors — is a lot of manual overhead to repeat every release. Safeguard brings structure to this process across your software supply chain, including the AI systems now embedded in it.
Safeguard helps teams:
- Maintain a living inventory of AI assets (models, agents, tool integrations, and their trust boundaries) so scoping a new exercise starts from an accurate picture instead of a fresh whiteboard session.
- Standardize attack taxonomies and severity rubrics across teams, so an LLM red team exercise run by one squad produces results comparable to another's.
- Automate evidence capture and reporting, turning red team transcripts and findings into audit-ready artifacts mapped to frameworks like OWASP LLM Top 10 and SOC 2 control requirements.
- Track remediation and regression status over time, flagging when a previously fixed issue resurfaces after a model or prompt change.
If AI adversarial testing is currently a one-time exercise your team runs before a big launch, Safeguard can help make it a continuous, auditable part of your supply chain security program — so every model update, prompt change, and new tool integration gets the same structured scrutiny as your code.