How do you evaluate an LLM for security work before trusting it?

Test it on your own data, not on a leaderboard. Build a small private set of real findings, measure how often the model is right and how often it is confidently wrong, add hostile inputs, and keep a human approving anything that changes code or access. Rerun the set whenever the model or prompt changes.

Why do public benchmarks not answer this?

A benchmark tells you how a model did on someone else's questions. Your questions have your languages, your frameworks and your noisy scanner output. NIST's Generative AI Profile (NIST-AI-600-1, released 26 July 2024) exists to help organizations identify risks specific to generative AI and choose actions that fit their own goals. The same idea works at team scale: measure against your use.

What should a security eval set contain?

Start with thirty to fifty cases, drawn from work your team already did. Mix them on purpose:

• Real vulnerabilities with the correct fix, so you can score triage and remediation advice.
• False positives your team closed, so you can see whether the model also closes them.
• Code that looks dangerous and is not, such as a sanitized query that a naive reviewer flags.
• Code that looks fine and is not.
• Questions with no answer in the supplied context, where the right output is "I cannot tell."

Keep expected answers in a file the model never sees. Write them before you run the model, not after.

Which scores are worth tracking?

Track a few numbers and ignore the rest. Accuracy on the confirmed cases is the obvious one. The more useful ones are the rate of confident wrong answers, the rate of invented package names or CVE identifiers, and consistency: the same input run five times should not give five verdicts.

If a model cites a source, check that the source exists and says what the model claims. OWASP's guidance on prompt injection recommends asking for reasoning and citations and using deterministic code to check output formats, which fits here.

Should you test it against hostile input?

Yes, defensively. Your model will read repository files, tickets and web pages that someone else wrote. OWASP lists prompt injection as LLM01 in its 2025 Top 10 for LLM applications. Its description says the input can be invisible to humans and that retrieval and fine tuning do not fully prevent it.

So add cases where a code comment or README tells the model to ignore its task, then check that the model's actions did not change. Test what the tool is allowed to do, not only what it says. Treat the model as an untrusted user, as OWASP suggests.

What does a failed run teach you?

Read the misses by type. If the model invents identifiers, restrict it to answer from supplied data and verify every identifier in code. If it misses real issues in one language, narrow its scope to the languages where it scores well. If results swing between runs, lower the randomness setting where your provider allows it, or use the model for drafting only and let deterministic scanners decide.

A model that fails your set for one task may pass for another. Evaluate tasks, not models in general.

What to do

1. Pick the one task you want the model to do, such as triage or fix suggestions. Evaluate that task only.
2. Build the private set of thirty to fifty cases above and write the answers first.
3. Run it, and record accuracy, confident errors, invented identifiers and run to run variance.
4. Add a handful of hostile input cases and confirm the model has no way to act on them without approval.
5. Set a pass line in advance. If the model misses it, change the scope, not the line.
6. Restrict the model to least privilege: read access, its own tokens, no standing write access.
7. Rerun the set on every model, prompt or tool change, and keep the results.

How does this connect to Safeguard?

Safeguard lets you choose Full AI, Zero AI or bring your own model, so you can run the same eval against each and compare. In Zero AI mode the scanners run without an LLM, which gives you a baseline to judge whether the model adds anything.

Quick check

Sources

• OWASP, LLM01:2025 Prompt Injection
• OWASP Top 10 for LLM Applications
• NIST AI Risk Management Framework

Last reviewed: 18 October 2026.