Create a throwaway repository, add one dependency version with a well documented advisory, run your scanner, and see whether it reports it. If it does, that path works. If it does not, you have found a coverage gap before an attacker or auditor does. Never deploy the test repository.
Why a canary test works
You cannot measure false negatives by looking at findings, because a miss produces nothing to look at. A planted canary gives you a known answer to compare against. It is the same idea as a smoke test in software: cheap, repeatable, and decisive when it fails.
Pick a canary with a documented advisory
Choose a component whose affected range is published by its own maintainers. The Apache Logging Services security page lists CVE-2021-44228 for log4j-core with affected ranges that include [2.13.0, 2.15.0). Version 2.14.1 falls inside that range, so it is a reasonable canary for a Java project. Confirm the range on the page yourself before relying on it, as ranges can be updated.
For other ecosystems, pick any package with a maintainer published advisory and a clearly affected version.
Steps
- Create an empty repository named something obvious, such as
scanner-canary. - Add a minimal manifest in the ecosystem you want to test, with exactly one planted dependency.
- Commit the lockfile too, if the ecosystem has one.
- Run the scanner the same way your pipeline does, with the same flags and credentials.
- Check whether the planted advisory appears, with the right version and package name.
- Repeat with the dependency as a transitive one, pulled in by another package.
- Record the result in a table: ecosystem, direct pass or fail, transitive pass or fail, date, scanner version.
Reading the result
| Result | Meaning | Next step |
|---|---|---|
| Found, correct version | Path works | Repeat for other ecosystems |
| Found, wrong version | Version resolution problem | Check lockfile handling |
| Not found, direct | Ecosystem or matching gap | Ask vendor, add a workaround |
| Found direct, missed transitive | Depth gap | Scan the resolved tree |
Safety rules
- Keep the repository private and never build or deploy it.
- Do not run the planted code. You are testing the scanner, not the library.
- Delete or archive the repository when done, so the planted finding does not enter your real metrics.
- Tag it so reports exclude it.
Repeat the test whenever you change scanner version, add an ecosystem, or change how pipelines invoke the tool. See also a clean scan usually means the scanner did not look and check scanner language and ecosystem support before choosing.
Try it yourself
Click through it at your own pace, or open it full screen. Watch how a finding moves from detection to a tracked item, which is what your canary should trigger.
Safeguard offers dependency scanning and a free scan at gold.safeguard.sh/scan. Run your canary through any tool you evaluate.
Quick check
How often should I rerun the canary?
After any scanner, ecosystem or pipeline change, and on a regular schedule such as quarterly.
Is it safe to add a vulnerable version to a repository?
In an isolated, never deployed repository, yes. Do not build, run or publish it.
Sources
Last reviewed: 19 October 2026.
