Take a sample of ten findings per scanner, apply each suggested upgrade on a throwaway branch, run your normal build and tests, and record whether it installs, compiles and passes. The share that works without further changes is a practical measure of how useful each tool's fix advice is.
Why test fixes at all
Most comparisons stop at detection. Finding a vulnerability is half the job, and the other half is closing it. If the advice is "upgrade to 3.4.1" and that upgrade breaks your build, or does not remove the finding, the tool has created work, not saved it.
Fix advice also differs between tools more than people expect. One may suggest the lowest version that clears the advisory, another the latest release, and a third may suggest a version that is only fixed in a different branch. Testing on your code shows which style suits you. See how a scanner chooses the fix version for the underlying logic.
Build the sample
From your adjudicated true positives, choose ten per scanner, or fewer if the set is small. Mix them:
- Direct and transitive dependencies.
- Patch, minor and major version jumps.
- Production and development packages.
Use the same selection rule for every tool, such as every Nth finding after sorting, so nobody can pick easy ones for a favorite.
The test procedure
- Start from the frozen commit and create a branch per finding. One finding per branch keeps cause and effect clear.
- Apply the suggested change exactly as the tool says. If it gives only a version, change the manifest and regenerate the lockfile the normal way.
- Install in a clean environment.
- Run your build, your unit tests and any quick integration tests.
- Rescan the branch with the same tool and with the other candidates.
- Record the outcome.
| Outcome | Meaning |
|---|---|
| Clean | Installs, tests pass, finding gone |
| Builds, finding remains | The suggested version does not actually fix it, or another path pulls the old version |
| Installs, tests fail | Breaking change needs code work |
| Does not install | Conflict with other constraints |
| No suggestion | The tool gave no version |
Score fairly
Report counts for each outcome per scanner, with denominators. A tool that suggests small patch bumps will naturally have more clean outcomes than one that suggests a major version, and that is a real property, but note the version jump size next to each row so readers can see it.
Treat "no suggestion" as its own category. A tool that says nothing about fixes for transitive dependencies is leaving that analysis to you.
Also check the rescan column. Occasionally a scanner suggests a version and still reports the finding after the upgrade because of a second copy of the package elsewhere in the tree. That mismatch is worth knowing about. The related post on findings that remain after an upgrade explains common causes.
Add effort to the picture
Record minutes per fix, from opening the finding to a green build. Include time reading the advice. If the tool opens a pull request for you, record whether it passed your checks without edits. Automated fix pull requests need care before you turn them on, as covered in evaluating automated fix pull requests.
Try it yourself
Click through it at your own pace, or open it full screen. Look at how a fix is presented next to the finding.
Safeguard's autonomous remediation can open fix pull requests, off by default with every category on Manual, so you decide what runs. The free scan at gold.safeguard.sh/scan shows findings you can use for this test.
Quick check
Do I need to test all findings?
No. A sample of ten per tool, chosen by a fixed rule, shows the pattern.
What if the build was already failing?
Fix the baseline first, or the test measures noise. Record the baseline result on the frozen commit.
Last reviewed: 19 October 2026.
