Date every result, finish within a short window, and rerun the most important tests just before signing. New advisories land daily and tools ship updates, so a comparison from two months ago describes tools that no longer behave the same way.
What ages in a comparison
Several inputs move while you decide.
- Advisories. New records appear, and existing records are modified. The OSV schema gives every record a modified timestamp for exactly that reason. See the schema.
- The tools. Releases change parsers, data sources and defaults.
- Your code. Dependencies get bumped and new repositories appear.
- The vendors. Packaging, features and support evolve.
None of this means comparisons are useless. It means they have a shelf life, which you should state.
Keep the window short
Plan the active part of the test to fit in a few weeks, with all candidates scanned on the same days. A rough pattern:
| Week | Activity |
|---|---|
| 1 | Freeze repositories, agree the rule, set up tools |
| 2 | Main runs, inventories and findings |
| 3 | Adjudication, fix tests, gate replay |
| 4 | Rerun key tests, write the readout |
The point of a rerun is to see stability. If a tool's findings on the same frozen commit changed between week 2 and week 4, find out whether the data changed or the tool did. The general method is in testing whether a scanner gives the same results twice.
What to rerun
You do not need to repeat everything. Rerun:
- One small repository per ecosystem, with every tool, using the saved commands.
- Your planted or back test cases.
- Any test where the two leading tools were close.
Compare the new run to the old one. Differences in findings should be explainable by new or edited advisories. Unexplained differences are findings in themselves about stability.
Record versions and dates everywhere
Every table in your readout should carry a run date and tool version. The spreadsheet columns post shows where. When someone reads the report in six months, they can see that the numbers describe October 2026 and that tool versions were as listed.
Write an expiry into the conclusion
End the report with a short statement such as: "These results describe the tools as tested on these dates. Re-check coverage for the ecosystems you rely on if the decision is revisited after a set period." Name a review date. If a purchase decision is delayed past it, rerun the quick set.
A decision rule written beforehand, as in the decision rule post, can include the maximum age of results.
After you choose
Continue to watch. Findings change after you adopt a tool too, because dependencies and advisories keep moving. A clean scan at build time is a snapshot, which is why continuous rescans matter. Keep your frozen test set and rerun it after major tool upgrades to catch regressions early.
Try it yourself
Click through it at your own pace, or open it full screen. Notice how a finding can appear later on code that did not change.
Safeguard's continuous monitoring rescans as new advisories arrive, and the free scan at gold.safeguard.sh/scan gives you a dated snapshot to add to your log.
Quick check
How old is too old for a comparison?
There is no fixed number. Set a review date in advance, and rerun the key tests if you pass it.
Do I rerun everything?
No. Rerun one small repository per ecosystem, your known cases and any close contests.
Sources
- OSV schema: each record has a modified field recording when it last changed.
Last reviewed: 19 October 2026.
