To measure what an SCA tool misses, create a test repository with dependencies you know are vulnerable, spread across direct, transitive and lockfile-only cases, then count how many the tool reports. Missed findings are invisible in normal use, so a seeded test is a direct way to see them.
Why false negatives are the hard part
Noise is easy to notice because someone has to read it. Missed findings are silent. A tool that reports few results can look clean and quiet, and nobody learns it skipped a manifest until an incident.
Evaluations that only count findings reward verbose tools. You want to measure recall too: of the vulnerable packages that are really there, how many did it see?
Build the test repository
Use a throwaway repository, not production code. For each ecosystem you ship, add:
- A direct dependency at a version with a published advisory
- A transitive dependency pulled in by another package, with a known affected version
- A package that appears only in a lockfile
- A package declared in a nested directory, as in a monorepo
- A development dependency
- A package from a private or mirrored registry, if you use one
Pick advisories from public sources and write down the package, version and advisory identifier for each. Avoid packages with disputed or withdrawn advisories, which confuse the result.
Run and record
Scan with the tool configured as you would in production. Record a table of seeded item, found or missed, how it was described, and which version it matched. Repeat the run with the same repository a few days later to see whether the database changed.
Check also for partial credit. A tool may report the package but assign the wrong affected range, or report it at a different version than the one installed. A wrong match is its own kind of miss.
Interpret honestly
A small test cannot give a statistic. Ten seeded items tell you where the gaps are, not the percentage you will experience across thousands of packages. Treat misses as leads: ask the vendor why, and see whether it is a configuration problem, a format you did not enable, or a real limit.
Look for patterns. If every miss is a lockfile-only case, you learned something specific about how it discovers dependencies. If every miss is in one ecosystem, you have a coverage answer.
Try it yourself
Open the demo and inspect the inventory view. Check that direct and transitive packages appear with versions, because that is the baseline your seeded test depends on.
Click through it at your own pace, or open it full screen.
Where Safeguard fits: Safeguard provides SBOM and dependency scanning. Point the free scan at gold.safeguard.sh/scan at your seeded repository and compare it with your other candidates.
Quick check
Is it safe to commit known vulnerable packages to a test repository?
Keep it private, mark it as a test, and never deploy it. The risk is low as long as the code never runs in production.
Last reviewed: 19 October 2026.
