Pick one repository you understand, run each tool, and review the same sample of findings by hand. Record true positives, false positives, and anything the tool missed. Two numbers fall out: how many of its findings were right, and how much real risk it failed to report. Vendor claims cannot give you those.
Choose the test repository
Use a service that has real dependency depth, ideally two ecosystems, and a maintainer who can explain why a package is there. Avoid a toy project. A repository with a lockfile and a few years of history is best.
Freeze it. Pin a commit so every tool scans identical code.
Build a small ground truth
Before running anything, take thirty direct and transitive dependencies and check them by hand. For each one, note the installed version from the lockfile and whether any published advisory affects that version. The npm lockfile docs describe the file as the exact tree generated, which makes it a fair reference for installed versions.
This is slow, and it is the only part that makes your numbers real. Do it for a sample, not everything.
Run the tools and label findings
Export each tool's findings. For a sample of fifty, label each one:
- True positive: the package and version are really present and really affected.
- False positive, wrong match: the tool matched the wrong package or an unaffected version.
- False positive, not applicable: the match is right, but the vulnerable code cannot run in your use. Count these separately, because they are a noise problem more than an error.
- Missed: something in your ground truth that the tool did not report.
Compute two simple numbers
Precision is true positives divided by all findings you reviewed. Recall is true positives divided by true positives plus misses in your ground truth sample. With samples this small, treat the numbers as rough. A gap of ten points between tools may be noise. A gap of forty is not.
Watch for the usual causes
Wrong matches often come from name collisions across ecosystems or from data that keys on loose identifiers instead of exact package versions. Misses often come from lockfiles the tool does not parse, vendored code, or private packages. Write down the cause for each miss, because it tells you whether the problem is fixable by configuration.
Compare time as well as accuracy
Note how long triage took per tool. A tool with slightly lower precision but a clear reason attached to each finding may cost less of your time.
For testing the noise side more directly, see how to test reachability claims.
Safeguard's dependency scanning with reachability analysis is one of the tools you can include in a pilot, and the free scan at https://gold.safeguard.sh/scan is a quick way to start.
Quick check
How big should the sample be?
Fifty findings and thirty components is a workable minimum. Larger is better if the team has time.
Is a not applicable finding a false positive?
It is a correct match with low practical risk. Count it separately so you can judge noise apart from errors.
Sources
Last reviewed: 19 October 2026.
