Scan the same commit three or more times, on different days, and diff the exported results. Some differences are legitimate, because advisory data changes. Others are not, such as findings that appear and vanish with no data change. A tool that cannot explain its differences is hard to trust in a pipeline.
Why repeatability matters
If a pull request passes at noon and fails at three with no change to the code, developers lose confidence and start rerunning pipelines until they get green. Auditors also ask whether a scan result can be reproduced. A tool that gives unstable answers undermines both.
There are two kinds of change to separate. The first is expected: new advisories land, so a package that was clean yesterday has a finding today. The second is noise: the same inputs produce a different output.
The protocol
- Fix the input. Choose one commit of a repository, and record its hash. Use a lockfile if one exists.
- Run it back to back. Scan three times in one hour, with the same settings. Export the full results in a machine readable format.
- Run it across time. Repeat the scan the next day and a week later on the same commit.
- Diff the exports. Compare findings by package, version and identifier, ignoring timestamps.
- Classify each difference. Mark it as new data, changed severity, or unexplained.
- Ask the vendor about the unexplained ones. A clear reason, such as a rule update, is a good sign.
Keep the commit and the exports in your trial folder.
What to compare
| Item | Expected to change? | Notes |
|---|---|---|
| Findings tied to new advisories | Yes | Check advisory dates |
| Severity or score of an existing finding | Sometimes | Ask what changed and why |
| Priority or reachability result | Rarely | Same code should give same answer |
| Set of detected packages | No | Same lockfile, same list |
| Order of results | Possibly | Compare as sets |
| Scan duration | Yes | Note outliers |
If the detected package list changes between runs on the same commit, ask hard questions. That points to flaky parsing or resolution.
Check the AI parts separately
If the product uses a language model for any step, such as explaining a finding or choosing a fix, ask whether that step is deterministic, and whether it can be pinned to a version. Run those steps three times and compare. Wording will differ. Conclusions, such as whether a finding is exploitable, should not flip without a reason.
Version pinning
Ask whether you can pin the scanner and its rule set to a version so a scan in March can be reproduced in September. If rules update continuously, ask how you can recover the rule version used for a past scan. That answer matters for audits.
Where Safeguard fits
Use this protocol on Safeguard like any tool. A convenient first run is the free scan at gold.safeguard.sh/scan, repeated on the same manifest over several days, with the exports compared.
Quick check
How many repeat runs are enough?
Three back to back and one a week later covers most cases. Add more if the first set shows differences.
Should results be identical after a week?
Not exactly. New advisories change findings. Differences should map to dated data changes.
Last reviewed: 19 October 2026.
