Run ten small tests against a throwaway repository, one per way a scanner can miss something, and record pass or fail for each. The result is a coverage map you can show an auditor and a plan for what to patch around. Rerun it whenever tooling changes.
How to use it
Create a private test repository that is never deployed. Each case adds one fixture, you scan, and you compare the result to the pass rule. Use dependencies with maintainer published advisories as fixtures. Record the date and scanner version beside each result.
The scorecard
| # | Case | Pass when |
|---|---|---|
| 1 | Direct dependency in a supported ecosystem | Advisory reported with correct version |
| 2 | Transitive dependency, two levels down | Advisory reported, with the path |
| 3 | Lockfile only, no manifest change | Version read from the lockfile |
| 4 | Version set by a parent or BOM | Resolved version reported |
| 5 | Dependency pinned to a git commit | Either reported, or listed as unresolved |
| 6 | Vendored copy with no manifest | Detected, or flagged as unidentified file |
| 7 | Checked in binary such as a JAR | Detected, or listed in inventory |
| 8 | Library inside a container image layer | Reported by the image scan |
| 9 | Nested project in a subfolder | Scanned, or listed as skipped |
| 10 | Internal package from a private registry | Resolved, or listed as unresolved |
Notice that several pass rules accept an honest "unresolved". A scanner that says "I could not identify this" is behaving well. One that says nothing is the failure.
Scoring
Count passes out of ten, but read the failures individually. Two failures in cases you never use may matter less than one in your main ecosystem. Weight by how much of your code each case represents.
What to do with failures
| Failure | Typical workaround |
|---|---|
| Transitive missed | Scan the resolved dependency tree |
| Vendored missed | Third party inventory file in the SBOM |
| Private registry unresolved | Fix registry credentials for the scanner |
| Nested project skipped | Configure scan roots |
Details for individual cases are in plant a known vulnerable package to test scanner coverage, test whether your scanner finds a vendored library and find manifests your scanner silently skipped.
Keep the evidence
Store the filled scorecard with your scan coverage statement. It shows exactly what was tested, rather than claiming coverage in general terms.
Try it yourself
Click through it at your own pace, or open it full screen. Compare the inventory it shows with the ten cases above and think about which would appear.
Safeguard offers SBOM and dependency scanning and a free scan at gold.safeguard.sh/scan. Run this scorecard on any tool you evaluate.
Quick check
How long does the scorecard take?
Setting up the repository takes a few hours once. Reruns are quick if you keep the fixtures.
Is ten out of ten realistic?
Not usually. The goal is to know where the gaps are, not to score perfectly.
Last reviewed: 19 October 2026.
