To test the fix loop, pick three findings, upgrade each package to the suggested version, run your tests, rescan, and check that the finding closes with no new ones introduced. A scanner that cannot close the loop cleanly will cost developers time long after the PoC.
Why test the loop
Finding a vulnerability is half the job. The other half is making it go away and being able to prove it. Developers judge a scanner on that second half. A tool that reports a problem, suggests an upgrade that breaks the build, then keeps flagging the package after the fix is a tool they stop opening.
The five step loop test
- Choose three findings: one direct dependency, one transitive, and one with a major version jump suggested.
- Apply the suggestion on a branch, using your normal package manager so the lockfile updates.
- Run your tests and build. Record pass or fail and what broke.
- Rescan the branch.
- Check the result: did the finding close, did new findings appear, did the scan's inventory show the new version?
Log it in a table.
| Finding | Suggested fix | Build and tests | Closed on rescan | New findings |
|---|---|---|---|---|
| Direct | Patch bump | Pass | Yes | None |
| Transitive | Parent upgrade | Fail | n/a | n/a |
| Major bump | Major upgrade | Pass after edits | Yes | One |
What to look for
Suggestion quality. Does the scanner name a specific fixed version, or only say upgrade? Does it say what to change for a transitive finding: which direct dependency to bump, or an override? If the only advice is to upgrade a package you do not control, that is where fix paths usually fail.
Build safety. A patch bump should rarely break anything. A major bump often will. Good tools flag the difference. If the fix breaks tests, note whether the scanner warned you.
Closure. After the fix, the finding should disappear on the next scan, quickly. If it lingers, find out why. Common reasons are a stale cache, a lockfile that was not updated, or a second copy of the package at another version. See the case study in why a scanner still flags an upgraded package.
No regressions. A new finding appearing after an upgrade is not a failure by itself. A newer version can pull in new transitive packages. But the report should make that visible.
Automated fix pull requests
If the product offers automated fix pull requests, treat that as a separate test and keep it off in the PoC unless you opt in. Run it on one repo, with human review of every change. Our two week plan for automated fix PRs covers it. Never let a trial tool merge on its own.
Turn it into a pass mark
A workable criterion is that most suggested fixes in your sample build, pass tests and close on rescan, and that every failure comes with a clear reason. Choose the threshold with your team and write it before the test.
Capture the time
Note how long each fix took from reading the finding to a green rescan. Two minutes for a patch bump and two hours for a major bump are both useful data points when you estimate rollout cost.
Try it yourself
Click through the demo to see how a finding shows its fixed version, then try the loop on your own repo with the same three steps.
Click through it at your own pace, or open it full screen.
Quick check
What if the fix breaks our tests?
Record it, note whether the scanner warned about it, and pick the next finding. Do not count a breaking fix as a pass.
Should we test fixes on production branches?
No. Use a throwaway branch and never merge trial changes without normal review.
Safeguard offers SBOM and dependency scanning with reachability analysis, and you can try a free scan at gold.safeguard.sh/scan before you plan a longer test. The CLI, desktop app and IDE extensions are listed on safeguard.sh/download.
Last reviewed: 19 October 2026.
