Evaluate an LLM for security work by running it on your own tasks, with answers you labeled in advance, several times each, and a record of the failures. Public benchmarks tell you little about your codebase, your advisories or your triage rules. The most common mistakes are in how teams build the test, not which model they pick.
What counts as a security task?
Be specific. "Security work" is a bundle: explaining a CVE, deciding if a finding is a false positive, writing a fix, summarizing a pull request for risk, mapping controls to a framework, or drafting an incident note. Each has a different failure cost and a different way to score. A model can be excellent at one and unreliable at another.
NIST's AI Risk Management Framework, a voluntary framework released on 26 January 2023, organizes risk work into Govern, Map, Measure and Manage. The Measure idea is the relevant one here: choose metrics tied to the context of use, and track them over time. NIST also published a Generative AI Profile on 26 July 2024. Use them as vocabulary, not as a test plan.
Mistake 1: trusting a general benchmark
A model that scores well on a coding or exam style benchmark may still mislabel your reachable findings. Build a set from your history: 50 to 100 past findings with the outcome your team agreed on, including the awkward ones.
Mistake 2: no ground truth, only vibes
If two senior engineers disagree on an item, it is not ground truth. Have two people label independently, resolve disagreements, and drop items that stay ambiguous. Keep a small set where the right answer is "not enough information", because a model that never says that is dangerous.
Mistake 3: one run per prompt
Outputs vary. Run each item several times and look at the spread. If the same finding gets "exploitable" on one run and "not exploitable" on the next, the average score hides the problem.
Mistake 4: scoring only accuracy
Track the cost of each kind of error. A missed critical issue and a spurious alarm are not equal. Also track refusals, formatting failures, invented package names or CVE numbers, and how often the model cites evidence you can check. Require sources for factual claims and test whether the cited item actually says what the model claims.
Mistake 5: ignoring the tool and data path
Most security work through an LLM runs with tools: file reads, scanners, tickets. OWASP's guidance on excessive agency says to minimize extensions and permissions and require human approval for high impact actions. Evaluate the model inside the real harness, with permission limits in place, and include tests where untrusted text in a file tries to steer the model. OWASP's prompt injection page says it is unclear whether foolproof prevention exists, and recommends treating the model as an untrusted user in testing.
Mistake 6: testing once, then forgetting
Models change, prompts change, your data changes. Keep the labeled set under version control, rerun it on every model or prompt change, and keep a changelog of scores. Add every real failure from production as a new test item.
How do you decide?
Set thresholds before you look at results: for example, no missed critical items on the labeled set, and a stated maximum rate of invented references. Then decide where a human stays in the loop regardless of score. For decisions with legal or safety weight, the score is an input to a person, not a replacement.
If you want flexibility, choose tooling that lets you swap models, so the evaluation result can change your choice without rebuilding the workflow. Safeguard supports Full AI, Zero AI or bring your own model, which makes a side by side test possible.
What to do
- List the three security tasks where an LLM would save the most time.
- Build a labeled set per task from your own history, with two independent labelers.
- Include unanswerable and adversarial items.
- Run each item several times and record the spread.
- Score error cost, invented references and format failures, not only accuracy.
- Evaluate inside the real tool harness with permission limits.
- Rerun on every change and add production failures to the set.
Quick check
How many test items do I need?
Start with 50 to 100 per task. It is enough to find gross problems, and you can grow it over time.
Can the model grade its own answers?
It can help with format checks, but verify its grading against human labels before trusting it.
Sources
Last reviewed: 18 October 2026.
