Measure scanner noise by sampling a fixed number of findings from each tool, having two engineers label each as actionable, not applicable or wrong, and comparing the share in each bucket. Count the time it took to decide, because the cost of noise is attention, not alert count.
Why raw counts mislead
A tool that reports four thousand findings is not worse than one that reports four hundred. Perhaps it is just more complete, or perhaps it duplicates. Raw counts cannot tell you. What matters is how many findings a person must read to find one worth acting on.
The sampling method
Run each tool on the same repositories. Randomly choose fifty findings from each result, using a script rather than your own picking. Hide the tool name if you can. Two engineers label each finding independently: actionable now, real but low priority, not applicable, or incorrect. Record the minutes spent. Resolve disagreements by discussion and note the reasons.
What to compute
For each tool, compute the share actionable, the share incorrect, and the median minutes per decision. Also check a few known true issues you planted or already know about, to measure misses. Noise is a trade against misses, so look at both. A quiet tool that misses a known issue has a different problem than a loud one.
Bring in exploitability context
Ask what extra context the tool gives: whether a known exploit exists, whether a fix is available, whether the vulnerable code is called. These can change labeling quickly, so compare tools with and without that context turned on. If the context is hard to see in the interface, count that too, because engineers will not click through five screens.
Keep the exercise honest by publishing your labels. If a vendor disagrees with a label, let them argue it with evidence, since sometimes they are right and your engineers missed a path. That debate is itself informative: you learn how the vendor handles disagreement and how clearly the tool explains its findings. Repeat the sample on a second repository to check that the result was not an accident of one codebase.
Try it yourself
Click through the demo and watch how the results change as you move through each step.
Click through it at your own pace, or open it full screen.
Where Safeguard fits
Safeguard offers vulnerability management with exploitability signals and reachability analysis, and a free scan provides output to sample.
Quick check
Is fifty findings really enough?
It is enough to see clear differences. For close results, increase the sample or repeat on another repository.
Last reviewed: 19 October 2026.
