How do you back test scanners with an old commit and a known advisory?

Find a past commit in your own history where a dependency was later found vulnerable and fixed, check it out, and scan it with every candidate. You already know the right answer, so you can score each tool's recall on real code without planting anything artificial.

The idea

Your repository history holds ground truth. At some point a dependency bump pull request fixed a vulnerability, perhaps one your current scanner reported or a customer asked about. The commit before that bump contained the vulnerable version. That is a labeled example from your own code.

Planting a known vulnerable package is a good test, as described in plant a known vulnerable package. A back test adds realism, because the dependency sits in your real build, with your real lockfile and layout.

Step 1: mine your history

Search for commits that changed dependency versions with a security motive. Useful clues:

• Commit messages mentioning security, advisory or a CVE.
• Pull requests opened by a dependency update bot with a security label.
• Tickets closed as fixed by an upgrade.
• Release notes that list dependency updates for security reasons.

For each candidate, record the commit before the fix, the package, the old version and the fixed version. Verify against the advisory that the old version really was in the affected range. A public vulnerability database lists affected ranges and fixed versions per record, so you can check yourself.

Aim for five to ten cases, covering different ecosystems and a mix of direct and transitive packages.

Step 2: build the test cases

Check out each commit to a clean directory and keep it frozen, as in the freeze procedure.

Step 3: scan and score

Run every candidate on each case with matched settings. Record, per tool and case:

• Was the vulnerable package identified in the inventory?
• Was the advisory reported?
• Did it report the correct fixed version?
• Did it report the same issue under other IDs, and were those grouped?

Score hits and misses with the denominator visible, such as "caught 6 of 8 cases." A missed case should send you to the inventory first: did the tool even see the package? That sequence is explained in comparing inventories before vulnerabilities.

Cautions

Data drift. Advisories published after a commit still apply to it today, which is what you want. But a case where the advisory was withdrawn or edited since can confuse results. Check the record's current state.

Survivorship. You only know about vulnerabilities that were noticed and fixed. Issues that nobody found remain invisible, so your test measures recall on known cases, not on everything. That limit is shared by all tests of this kind, as discussed in what no single scanner finds.

Small sample. Eight cases cannot support a precise recall rate. Use it to find clear misses and to compare tools qualitatively.

Same story for every tool. Do not tell any vendor which cases you chose. Use some as a hold-out, per the hold-out post.

What a miss teaches

If a tool misses a case, find out why. Common causes include an unsupported lockfile format at that point in history, a vendored copy, or a package source the tool does not know. Each cause tells you something about fit with your codebase. Another reason could simply be that the tool's data lacks the advisory, which matters more.

Safeguard note

Safeguard's dependency scanning and zero-day discovery work on your repositories, and the free scan at gold.safeguard.sh/scan can be run on a checked out old commit to include in your scoring.

Quick check

Sources

• OSV schema: records list affected packages, ranges and fixed versions.

Last reviewed: 19 October 2026.