Evals measure how a system behaves on a fixed set of test cases, so you can repeat them and compare results over time. Red teaming is a structured attempt to find flaws you did not already know to test for, using an adversary's mindset. You need evals for regressions and red teaming for discovery.
How they differ
- Goal. Evals: Measure and track. Red teaming: Discover and break.
- Test set. Evals: Fixed, versioned. Red teaming: Evolving, creative.
- Run by. Evals: Automation in CI. Red teaming: People, often with automation.
- Cadence. Evals: Every change. Red teaming: Scheduled, or after big changes.
- Output. Evals: Scores and trends. Red teaming: Findings and attack paths.
NIST defines AI red-teaming in its Generative AI Profile as a structured testing exercise used to probe an AI system to find flaws and vulnerabilities such as inaccurate, harmful or discriminatory outputs, often in a controlled environment and in collaboration with system developers.
What evals are good at
They catch regressions. If a prompt change or a model upgrade makes the assistant leak a system prompt that it did not leak last week, a fixed test fails and a person finds out the same day.
They are cheap. Once written, a few hundred cases can run on every pull request.
They give a number you can put on a chart, which helps argue for a fix.
Their weakness is that they only test what someone thought to write. An attacker who thinks differently walks around them.
What red teaming is good at
It finds the cases nobody listed. A human tester notices that a support assistant retrieves from a shared folder that outsiders can write to, and builds an attack that no fixed test would include.
It covers the whole system. The ATLAS AI Red Team mitigation says exercises should include models, data, agents, memory, tools, retrieval systems, identities and permissions, not just prompts.
Its weakness is cost and repeatability. Two red teams can reach different findings, and a one-off exercise goes stale as the system changes.
How they work together
- Run a red team exercise. Record each successful attack.
- Turn each one into an eval case, so it cannot return unnoticed.
- Run the eval set on every change.
- Repeat the exercise when the system changes in a big way, with fresh testers if you can.
ATLAS makes the same loop explicit: assign findings to owners, retest corrected systems, and use demonstrated attacks to improve future preparation.
Common mistakes
- Treating a high eval score as proof of security. The score only reflects the cases you wrote.
- Treating a red team report as a permanent certificate. It describes one moment.
- Using a model to grade outputs without checking the grader. See LLM-as-judge pitfalls.
- Writing evals only for harmful content and none for data leaks, tool misuse or retrieval of the wrong user's data.
Quick check
If I have a good eval suite, do I still need red teaming?
Yes. The suite covers known cases. Red teaming looks for unknown ones, and its findings should feed your suite.
Which should a small team start with?
Start with a small eval set covering your top risks, then add a scoped red team exercise before a major launch.
Sources
- NIST AI 600-1, Generative AI Profile
- MITRE ATLAS data repository (mitre-atlas/atlas-data), mitigation AML.M0035, 2026.09 release.
Last reviewed: 19 October 2026.
