What are AI evals and how do you write useful ones?

AI evals are repeatable tests that measure how well a model, prompt or agent performs on tasks you care about. Each eval has a dataset of inputs, a way to run the system, and a scorer. Treat them like unit tests: build them from real cases and rerun them on every change.

Why do evals matter for security teams?

A model update, a prompt edit or a new tool can change behavior without any code diff you can read. Evals turn "it seems fine" into a number you can compare. For security work they answer questions such as: does the assistant still refuse to reveal secrets, does the agent stay within its tools, does the code reviewer still catch the injection flaw it caught last week.

The NIST AI Risk Management Framework has a function called measure, which calls for methods and metrics that you can repeat. Evals are the working form of that.

What are the parts of an eval?

The UK AI Security Institute's open framework Inspect describes composable building blocks: datasets, agents, tools and scorers. It ships with over 200 pre-built evaluations that can run on any model, and can assess coding, agentic tasks, reasoning, knowledge and behavior. You do not need that framework to benefit from the structure.

1. Dataset: inputs with expected outcomes or grading notes.
2. Task: how the system is run, including the prompt, tools and settings.
3. Scorer: how to judge the output. It can be exact match, a test that runs the generated code, a rule, or another model acting as grader.
4. Report: the score overall and per category, with failures you can read.

What makes an eval useful?

Realistic inputs. Cases pulled from your own tickets, logs and code beat invented puzzles. Include the weird ones.

Clear scoring. If two people disagree on whether an answer is right, the scorer will be noisy. Write the rubric first.

Coverage of failure. Add cases that should be refused, cases with injected instructions, and cases where the right answer is "I do not know."

Separation. Keep a held-out set that you never tune against. If you fix the prompt until every case passes, you have only learned the test.

Repeatability. Record the model name and version, settings and date, because results shift between runs and versions. Run each case several times when outputs vary.

What are the limits?

Public benchmarks measure public tasks. A high score does not show the model suits your data. Model-graded scoring inherits the grader's biases and can be fooled. And an eval measures what you thought to test: it cannot show the absence of unknown problems. Pair it with red-team exercises and monitoring in production.

A constructed example

A team builds an agent that triages vulnerability reports. They collect 120 past reports with the decision a senior engineer made: fix now, schedule, or dismiss. They hold out 30. Scoring has two parts: agreement with the engineer, and a hard rule that no report with a known-exploited issue may be dismissed. The first prompt agrees 71 percent of the time and fails the hard rule twice. They fix the prompt, rerun on the held-out set, and keep the suite in CI so the next model switch reports its score before anyone ships it.

How to check or apply this

1. List the behaviors that must not regress, and the ones that must improve.
2. Collect 50 to 200 real cases per behavior, and label them.
3. Write the scoring rubric, and test the scorer on a few cases by hand.
4. Include refusal, injection and "unknown" cases.
5. Hold out a set that no one tunes against.
6. Run evals in CI on every prompt, model or tool change.
7. Store results with model version and settings.
8. Read failures every week. The failures are where the learning is.

Quick check

Sources

• Inspect, UK AI Security Institute evaluation framework
• NIST AI Risk Management Framework
• OWASP GenAI Top 10 for LLM Applications

Last reviewed: 18 October 2026.