Give every finding that only one scanner reports a verdict from a short, fixed sheet: is the package really installed, is the version affected, does an authoritative record agree, and does the finding matter here. Record the verdict and the evidence, not just a yes or no.
Why this step decides the comparison
After you normalize findings, most will match. The interesting rows are the ones where scanner A speaks and scanner B is silent. Either A found something real that B missed, or A raised a false alarm. Counting these rows without judging them tells you nothing. Scoring them correctly is the core of a fair comparison.
For the join itself, see matching findings with purl and aliases.
The four question sheet
For each one sided finding, answer in order and stop at the first failure.
- Is the package present? Check the lockfile and, if unsure, the installed tree. If it is not there, the finding is a false positive, and the cause is an inventory error.
- Is the installed version inside the affected range? Read the advisory's affected ranges. If the version is outside, the finding is a false positive from matching.
- Does an independent record agree? Look up the advisory in a public database that is not either scanner's own. If it cannot be found, mark it unverified rather than false.
- Does it matter here? Is the vulnerable code used, is the package only in a test dependency, does a fix exist? This does not decide whether the finding is correct, but it decides priority.
Questions 1 to 3 judge correctness. Question 4 judges usefulness. Keep them separate in your records. A finding can be correct and still low priority.
A sheet you can copy
| Field | Example entry |
|---|---|
| Finding ID | Row from the joined table |
| Reported by | A, B, or both |
| Package present? | Yes or no, with lockfile line |
| Version affected? | Yes or no, with the affected range |
| Independent record? | Link or "not found" |
| Verdict | True positive, false positive, unverified |
| Priority note | Production, dev only, no fix, etc. |
| Reviewer and date | Name and date |
Keep this in a shared spreadsheet. A second person should re-review a sample, around one in five rows, to check you are applying the sheet the same way. If two reviewers disagree often, your questions are vague, not your scanners.
Scoring after adjudication
With verdicts in hand, tally per scanner:
- True positives it reported.
- False positives it reported.
- True positives it missed, meaning the other scanner found them and the verdict is true.
That gives precision and a lower bound on recall. It is a lower bound because both tools could miss the same real issue. The idea is covered in two scanners union or intersection and in capture recapture estimates.
Cases that need extra care
Unverified findings. If no independent record exists, do not count the finding as true or false. Report the unverified count separately. A scanner that surfaces many unverified items is making a claim you cannot check.
Distribution patches. A package patched by an operating system vendor may carry the same version string as an affected release. Verdicts here depend on the vendor's own advisory, so check it before calling a false positive.
Duplicate groups. If one tool reports three advisories for what is one flaw, adjudicate the group once.
Keep the reviewers blind where possible
If the reviewer knows which tool reported a finding, the verdict can drift toward the favorite. Strip tool names from the sheet until verdicts are filed. The blind triage post shows how.
Safeguard's reachability analysis can supply evidence for question 4, whether the vulnerable code is used. The free scan at gold.safeguard.sh/scan lets you add a result to the sheet.
Try it yourself
Click through it at your own pace, or open it full screen. Notice the evidence that helps answer whether a finding matters here.
Quick check
How many one sided findings must I review?
All of them if there are fewer than about fifty. Above that, review a random sample from each scanner and say how you sampled.
What if the reviewers disagree?
Tighten the question wording, rule on the disputed rows together, and re-review a sample.
Last reviewed: 19 October 2026.
