LLM Red Teaming Tools Compared: garak, PyRIT and More

The main open source LLM red teaming tools are garak, PyRIT, Promptfoo and DeepTeam. garak scans models with many probes, Promptfoo runs plugin based tests in CI, DeepTeam targets agents and RAG pipelines, and PyRIT is a Microsoft framework for security engineers. Choose by target and workflow, not by name.

Safeguard publishes this guide and sells a competing product, so we describe other tools only from their own documentation.

Key takeaways

• All four tools in this comparison are open source, and three state a license on their repository page.
• garak behaves like a scanner, while PyRIT is described as a framework you build with.
• Promptfoo and DeepTeam both state a path to repeatable, automated runs.
• Single prompt tests miss multi-turn attacks. DeepTeam lists multi-turn methods such as Crescendo jailbreaking.
• A red team tool finds problems. You still need to fix and retest.

What is LLM red teaming?

LLM red teaming means sending adversarial inputs to an LLM application and checking whether it fails in ways you care about. Promptfoo describes the loop as generating adversarial inputs, running them through the app, and analyzing outputs with deterministic and model graded metrics.

The target is your application, not only the base model. Your system prompt, retrieval data, tools and permissions change what can go wrong.

How do garak, PyRIT, Promptfoo and DeepTeam compare?

Facts below come from each project's own page, fetched as of 19 October 2026.

"Not stated" means the page we fetched did not say, not that the feature is absent. Read the project docs before deciding.

When does each tool fit?

• garak: you want a broad scan of a model or endpoint, with a report per run. By default it runs all known probes with each probe's recommended detectors, and you can limit scope.
• PyRIT: you are building a custom red teaming workflow and want a framework. Its repository now lives at microsoft/PyRIT, and an older Azure/PyRIT repository is archived, so link to the current one.
• Promptfoo: you want tests in a CI pipeline with plugin based coverage such as authorization flaws.
• DeepTeam: you test agents or RAG and want named agentic vulnerabilities and multi-turn attacks.

How do you run a first test?

A first garak run looks like this:

python -m pip install -U garak
garak --list_probes
garak --target_type openai --target_name <your-model> --spec probes.encoding

Replace the model name with your own and set the provider key in your environment first. Start with one probe family, read the FAIL rate, then widen.

Run these tests against staging, never against a production system holding real customer data.

How to choose

• List your targets: raw model, chat app, RAG, agent with tools.
• Confirm the tool lists those targets on its own page.
• Check for multi-turn support if your app keeps conversation state.
• Decide where results go: a JSONL file, a CI gate, a ticket.
• Check whether the tool needs a third party model key to judge results.
• Plan retests after each fix.

For where red teaming sits among other controls, read best AI security tools by layer.

Try it yourself

Safeguard offers a model risk scan demo and a free browser dependency scan at gold.safeguard.sh/scan for the packages your AI project imports. Red teaming covers behavior, and a dependency scan covers the code underneath.

Click through it at your own pace, or open it full screen.

Frequently asked questions

Sources

• garak on GitHub
• PyRIT on GitHub
• Promptfoo red teaming docs
• DeepTeam on GitHub
• OWASP Top 10 for LLM Applications

Last reviewed: 19 October 2026.