Evaluating an LLM for security work: the evals that matter

To evaluate an LLM for security work, build tests from your own tasks, with known answers, and measure accuracy, false positives, safety of suggested code and behavior when the model does not know. Rerun them on every model or prompt change. A public leaderboard score tells you little about whether the model helps your team.

What is an eval, in practice?

The UK AI Security Institute's open source Inspect framework gives a clean structure. An evaluation is a Task with three parts. The dataset holds labeled samples, usually an input and a target answer or grading guidance. The solver produces an answer for each sample, from a single model call to a full agent using tools. The scorer judges the output through text comparison, model grading or a custom method.

You can use Inspect or write a spreadsheet and a script. The structure is what matters: fixed inputs, a recorded target, a repeatable score.

Which evals matter for security tasks?

Pick the jobs you actually plan to give the model.

Triage accuracy. Give the model real findings from your past scans, with your team's final decisions as the target. Measure agreement, and look separately at false positives (noise that wastes time) and false negatives (real issues dismissed). A model that is right 90 percent of the time can still be unusable if its misses are the critical ones.

Explanation quality. For findings it explains, have an engineer rate whether each explanation is correct and actionable. Sample, and keep raters blind to the model.

Fix suggestions. For suggested patches, run the tests and the scanners. OWASP's misinformation entry lists unsafe code generation as a risk, including suggestions of insecure or nonexistent libraries. Check that suggested packages exist and that the fixed code passes your tests and scans.

Honesty about uncertainty. Include questions with no answer in the provided context. A good model says it does not know. A bad one invents.

Tool use. If the model drives tools, test that it stays inside scope and asks for approval where required. Score actions taken, not only text.

Consistency. Reword the same task several ways and check the answers stay consistent. For prompt injection, OWASP advises treating the model as an untrusted user and running regular adversarial testing, which you can do with your own safe test cases.

Cost and speed. Track tokens and time per task so you can compare options on real numbers.

How do you keep evals honest?

• Keep a held-out set you never use to tune prompts.
• Record model name, version, prompt and settings with each run.
• Document test sets, metrics and tools, as NIST's AI RMF suggests under Measure 2.1.
• Refresh samples as your codebase and threats change.
• Have a second person verify the target answers.

How does this relate to choosing a model?

If your platform lets you pick the model, evals let you choose with evidence. Safeguard supports Full AI, Zero AI and bring your own model, which includes on-prem and local options. An eval on your own tasks tells you whether a smaller local model is good enough for triage, or whether you need a larger one for fix writing. Zero AI is also an option for work that needs no model at all.

What to do

1. Write down the three security jobs you want the model to do.
2. Collect fifty real samples per job with final human decisions as the target.
3. Split off a held-out quarter and leave it alone.
4. Choose scorers: exact match where possible, human review where not.
5. Run each candidate model and compare accuracy, false positives, cost and refusal behavior.
6. Rerun on every model, prompt or tool change and keep the history.

Quick check

Sources

• Inspect, the UK AI Security Institute evaluation framework
• OWASP GenAI: LLM09:2025 Misinformation
• OWASP GenAI: LLM01 Prompt Injection
• NIST AIRC: AI RMF Core

Last reviewed: 18 October 2026.