When a repository has no lockfile, build it once, record the installed versions as your reference, and measure each scanner against that reference. Without it, tools will resolve version ranges differently and you will have no way to say who is closer to what actually ships.
Why missing lockfiles are a special case
A lockfile records the exact versions a build installs. A manifest with ranges, such as "any version at or above 2.1," does not. A scanner reading only the manifest has to choose: assume the lowest allowed version, resolve a version live from the registry, or skip the package. Each choice produces different findings, and on different days the live option can change by itself.
If your test set includes such repositories, and many older projects do, treat them as their own category.
Step 1: make a reference
Install dependencies in a clean environment from the frozen commit, using your normal build. Export what actually got installed. Most package managers can list installed versions in a machine readable form. Save that file with the date. This is your reference list, and it is only valid for that day, since a later install may pull newer versions. The freeze post explains why the date matters.
Optionally, commit the generated lockfile to your frozen test copy and give that to all scanners as a second test. You then have two scenarios.
| Scenario | Input to scanner | What it tests |
|---|---|---|
| A: as is | Manifest only | How the tool copes with ranges |
| B: with lockfile | Manifest plus generated lockfile | Baseline matching quality |
Step 2: scan both scenarios
Run every candidate on scenario A and scenario B. Record for each:
- Did it warn that no lockfile was found?
- Did it resolve versions, and does its documentation say how?
- How many packages did it report compared with the reference list?
- Did it report the same version as the reference for each package?
Some tools can resolve transitive dependencies from the manifest by contacting a registry, and some can be told not to. OSV-Scanner documents a flag that turns off transitive resolution, for example, which means the behavior is a setting you should state in your notes. See the usage page.
Step 3: grade against the reference, not each other
For each tool, count packages where the reported version equals the reference version, packages with a different version, and missing packages. Do this per ecosystem.
A tool that reports the lowest allowed version will flag older, more vulnerable releases that your build may never install. That can look like a gain in detection while actually being noise. A tool that resolves live may match the reference today and drift next week. Neither is wrong in a universal sense, but one may fit your build better.
Step 4: decide what you actually want
Ask the practical question: for a repository without a lockfile, what do you want a scanner to do? Common answers include:
- Warn loudly and ask for a lockfile.
- Resolve as the package manager would, and show the versions chosen.
- Report the range and flag any vulnerable version inside it.
Option 1 is often the most honest. Whichever you prefer, it may be a stronger reason to add lockfiles than any scanner feature. Fixing the repositories is a result of the comparison, not a distraction from it.
Common mistakes
- Treating a manifest only scan as equivalent to a lockfile scan.
- Letting one tool run a fresh install and another not.
- Counting the lowest version assumption as extra "coverage."
- Forgetting that the reference expires.
Safeguard's dependency scanning builds an inventory from your repositories, and the free scan at gold.safeguard.sh/scan is a quick way to see how a repository without a lockfile is handled before you set up a full test.
Try it yourself
Click through it at your own pace, or open it full screen. Look at the version each package resolves to and how you would check it against a reference.
Quick check
Should I add lockfiles before the test?
Test both ways. The as is scenario shows how tools cope with reality, and the lockfile scenario shows baseline quality.
Which tool is right when versions differ?
The one that matches what your build installs, which is why you create a reference first.
Sources
- OSV-Scanner usage: transitive resolution can be disabled by flag.
Last reviewed: 19 October 2026.
