To test AI vulnerability triage, build an answer key from findings your team has already judged, run the feature on the same findings without showing it the answers, and compare. Count agreement, missed real issues, and confident wrong calls. Twenty to fifty findings is enough for a first signal.
Step 1: build the answer key
Pull findings from the past year that engineers resolved with a clear outcome: fixed, not exploitable, accepted risk, or false positive. Record the reason for each. If the reasons are not written anywhere, ask the engineers now while they remember.
Aim for a mix. Include some high severity findings that turned out not to matter and some medium ones that did. Triage only gets interesting where severity labels mislead.
Step 2: hide the answers
Run the AI triage on the same findings and make sure it cannot see your past decisions. If the tool reads your ticket history, strip those comments for the test. Otherwise it may simply repeat what people already wrote.
Step 3: score three ways
| Measure | How to count it |
|---|---|
| Agreement | Same decision as your key |
| Dangerous miss | Called safe, key says it needed action |
| Noise | Called urgent, key says it did not matter |
The dangerous miss is the number to watch. A feature that lowers noise by hiding real issues is trading one problem for a worse one.
Step 4: ask for the reasoning
For ten findings, read the explanation. Does it point to something specific, such as whether the vulnerable function is called, whether a fix exists, or whether the component is in a deployed image? Or does it restate the CVE description in confident prose? Reasoning tied to your code is evidence. Generic prose is decoration.
Step 5: repeat on fresh data
A tool that scores well on historical cases may do differently on new ones. Run it on this month's findings, let your team triage them in parallel, and compare again. Two rounds beat one.
What a good result looks like
You do not need perfection. You need a result better than your current process on the measure that matters to you, with failures that are visible and recoverable. Agree on that threshold before you run the test.
Try it yourself
Click through it at your own pace, or open it full screen.
Look at how findings are prioritized and what information sits next to each one. That is the kind of context to compare against your answer key.
Where Safeguard fits
Safeguard includes reachability analysis and SBOM and dependency scanning, and the free scan at gold.safeguard.sh/scan lets you see prioritized results on your own project.
Quick check
What if we have no history of resolved findings?
Have two engineers triage twenty current findings independently and use their agreed answers as the key.
Is a vendor benchmark enough?
It is a starting point. Your code and dependencies are what you will run it on.
Last reviewed: 19 October 2026.
