Run each candidate three times on identical, frozen inputs and compare the outputs. If findings change between runs, you have found a repeatability problem, and no review score will tell you that. It takes an afternoon and prevents surprises in production.
What can vary
Several things can change between runs without any code change.
- Rule or vulnerability data updated between runs.
- Network lookups that time out or return different data.
- Parallel processing that orders results differently or drops items under load.
- Models or heuristics that produce different output on repeated calls.
- Build steps that resolve different dependency versions.
Some of this is acceptable. A vulnerability feed update that adds a new advisory is a feature. Unexplained loss of a finding is not.
How to run the test
- Freeze the commit and the build. Pin versions in a lockfile so the scan target does not move.
- Record the tool version, rule set date and settings.
- Run the scan three times, ideally at different times of day.
- Export results in a machine readable format.
- Diff the outputs: findings present in all three, in two, in one.
What to measure
- Count of findings in each run.
- Findings that appear in only some runs.
- Differences in severity or description for the same finding.
- Scan duration and its spread.
- Any run that failed or was partial.
If a tool reports a clean result on a run, check it is not silent failure. A scan that cannot reach its data source should say so loudly.
How to treat the results
Variation is not always a disqualifier. If a tool varies only when a data feed updates, document it and decide whether you can live with it. If it varies with no explanation, ask the vendor, and treat a vague answer as a risk. For gates that block merges, repeatability matters more, because a flaky block costs developer time.
Report it fairly
Apply the same test to every candidate and report the same measures. A reviewer who tests only one run per tool cannot say anything about this. Ask any review you read whether it did. See how to read a review critically.
Try it yourself
Click through it at your own pace, or open it full screen.
Look at how results change over time and ask which changes come from new data and which would count as instability.
Where Safeguard fits
Safeguard offers SBOM and dependency scanning, policy gates and continuous monitoring of dependencies. A free scan is available at gold.safeguard.sh/scan; run it more than once and diff it.
Quick check
Why not run the scan once and move on?
One run cannot show variation. Three runs can show whether results are stable before you rely on them in a pipeline.
Last reviewed: 19 October 2026.
