How do you freeze your code before comparing dependency scanners?

Pin each repository to a specific commit SHA before any scanner runs, give every candidate that exact checkout, and write the SHAs down. If one tool scans on Monday and another on Thursday, and the team merged a dependency bump in between, you are comparing code, not scanners.

Why a moving target ruins the comparison

Your repositories change every day. A merged pull request can add a package, remove one, or upgrade a vulnerable version. If two scanners see different states of the same repository, every difference in their results has at least two explanations, and you cannot tell them apart afterward.

A frozen input removes one whole class of arguments. When someone says "Scanner B missed this," the answer is checkable: same commit, same lockfile, same package, different result.

The freeze procedure

1. List the repositories in your test set. Aim for variety, not volume. A guide to picking them is in how to choose a scanner test corpus.
2. For each one, record the full commit SHA of the branch you are testing, not a branch name. Branch names move. SHAs do not.
3. Create a read-only copy or a tag per repository, for example a tag named scanner-compare-2026-10. Every candidate scans from that tag.
4. Record the date and time of each scan run, and the scanner version.
5. Do not let anyone merge dependency changes into the tagged state during the test window. Dependency bots will keep opening pull requests. Let them, but do not merge them into the frozen copy.

A small table keeps this honest:

What else needs to stay still

The code is only one input. A scanner also reads vulnerability data, and that data changes daily as new advisories land. You cannot freeze the world, but you can control for it two ways.

• Run all candidates within the same short window, ideally the same day, so each sees roughly the same advisory state.
• Record the database or data version each tool reports, if it reports one. If a tool will not tell you how fresh its data is, note that as a finding.

For tools that match against a public database such as OSV, remember each record has a modified timestamp. A record edited between two runs can flip a result. That is not a scanner defect, it is a data change, and your log should let you say so. See the OSV schema for how records are versioned.

Handling private dependencies and generated files

If your build generates a lockfile on the fly, freeze the generated lockfile too, by committing it to the tagged copy. Otherwise each scanner may resolve versions differently on the day it runs. Resolution differences are a real thing to test, and a separate post covers repositories without lockfiles. Do that as its own experiment, not by accident.

Prove the freeze afterward

At the end, check that the scanned state matches the record. Run a quick comparison of the SHA each tool reports, if it shows one, against your table. Some tools print the commit in their output or report metadata. If yours do not, keep your own run log. When you write up the results, include the SHA table as an appendix. Anyone who doubts the numbers can rerun them.

Safeguard's dependency scanning and SBOM generation work from your repositories, and the free scan at gold.safeguard.sh/scan is a low effort way to add a candidate to a frozen test. The freeze procedure above applies to it the same as any other tool.

Try it yourself

Click through it at your own pace, or open it full screen. Look at how one repository yields a package inventory you could record as your baseline.

Quick check

Sources

• OSV schema: records carry id and modified fields, which is why a record edit between runs can change a result.

Last reviewed: 19 October 2026.