Have two reviewers label the same 20 to 30 scanner findings independently as true, false or unclear, then compare. If they disagree often, your false positive rate is a matter of opinion, and you need clearer labeling rules before you trust any accuracy number from the PoC.
Why this matters
Most PoCs report a false positive rate based on one person's judgment. But triage is judgment. One engineer may call a finding false because the vulnerable function is not called. Another may call it true because the package is present at an affected version. Both are defensible. They are answering different questions.
If you do not check agreement, you may compare two scanners on two reviewers' moods.
Define the labels first
Write short definitions before anyone looks at a finding. A workable set:
| Label | Meaning |
|---|---|
| Accurate | Package is present at an affected version and the advisory applies |
| Inaccurate | Package or version is wrong, or the advisory does not apply |
| Accurate but not used | Present and affected, but vulnerable code is not reachable in your app |
| Unclear | Reviewer cannot decide with a few minutes of effort |
Separating "inaccurate" from "accurate but not used" is the key. The first is a detection error. The second is a prioritization question. Mixing them makes every scanner look worse than it is.
The method
- Draw a random sample of 20 to 30 findings, the same set for both reviewers.
- Hide the scanner's severity, or at least do not discuss it, to avoid anchoring.
- Each reviewer labels independently and notes a one line reason.
- Put the labels side by side.
- Count agreements, and for each disagreement, discuss and decide a final label.
Reading the result
A simple measure is the share of findings where both reviewers gave the same label. There are formal statistics for chance corrected agreement, but a plain percentage and a list of disagreements are enough for a PoC.
- High agreement: your labels are stable. Use them.
- Moderate agreement with a pattern: often one type of case, such as dev only packages, causes the splits. Add a rule to the definitions and relabel.
- Low agreement across the board: the criteria are unclear, or the findings lack context. Fix the definitions or ask the scanner for more detail.
What disagreements reveal about the scanner
Sometimes the disagreement is about the tool. If reviewers cannot decide because the output does not show the dependency path, the version, or the advisory text, that is a usability finding. Log it as such. A finding that two experts cannot judge in a few minutes is hard for a developer too.
Use reviewers with different roles
Pair a security reviewer with a developer who owns the code. Their disagreements are the interesting ones, because they mirror the real gap between security and engineering. Note which labels each tends to use, and design the workflow around that.
Keep it light
This costs about an hour per reviewer. Do it once mid trial, then use the settled labels for the final accuracy number in your memo. If you are comparing two scanners, give both the same reviewers and a blind sample, as in our post on blind scoring.
Try it yourself
Open the demo and look at how a finding can be present yet not exploitable, which is the exact split your labels need to capture.
Click through it at your own pace, or open it full screen.
Quick check
Is 20 findings enough?
It is enough to expose unclear labeling rules. It is not a statistical sample, so describe results as indicative.
Should reviewers talk first?
No. Label independently, then discuss. Talking first hides disagreement.
Safeguard offers SBOM and dependency scanning with reachability analysis, and you can try a free scan at gold.safeguard.sh/scan before you plan a longer test. The CLI, desktop app and IDE extensions are listed on safeguard.sh/download.
Last reviewed: 19 October 2026.
