Yes. Pick a dependency vulnerability your team already fixed, check out the commit from before the fix, and scan it in the demo. If the scanner flags the known problem, you have real evidence. If it misses it, you have learned something no brochure would tell you.
Why a past incident makes a good test
You already know the right answer. You know which package, which version, and roughly when the advisory appeared. That removes the usual problem in demos, where nobody can tell a true finding from a false one.
It is also relevant. A problem that hit your own codebase is more representative than a planted example from a vendor's test set.
Steps
- Find the fix commit. Search your history for the upgrade, or ask the engineer who did it. Note the package, the old version and the new version.
- Check out the parent commit. Create a branch from the commit just before the fix. Push it to a test location the vendor may scan.
- Write your expectation down. One line: package, old version, advisory identifier if you have it.
- Scan live. Ask the vendor to scan that branch without telling them what to look for.
- Compare. Did the tool report the package? Which advisory? Is the suggested fix the version you chose?
Reading the outcome
| Outcome | What it means | Next step |
|---|---|---|
| Found, same fix | Good match | Test a second incident |
| Found, different fix | Worth understanding | Ask why; both may be valid |
| Found at a different severity | Scoring differs | Ask which score is shown |
| Not found | Gap or data issue | Ask whether the advisory exists and why it was not matched |
A miss is not automatic disqualification. The advisory may have been published after your scan date, or the package may sit in a part of the repo the scan skipped. The point is to find out which, with the vendor in the room.
Pitfalls
- Time mismatch. Advisory data today includes records published after your fix. That favors the scanner, so a find proves less than a miss.
- Changed build. The old commit may not build on current tools. You do not need a successful build for a dependency scan, but you do for anything that needs compilation.
- Private package. If the incident involved an internal package, public advisory data will not cover it.
Make it a pattern
Keep a short list of two or three past incidents. Reuse them whenever you assess a new tool. Over time they become your own benchmark, built from your own history, and they cost nothing.
You can also add one thing to the replay: ask how long the finding would have been visible before the fix. If your team learned about the issue weeks after an advisory was published, a tool that raises it on the day is worth a lot.
Where Safeguard fits
Safeguard offers SBOM and dependency scanning, plus zero-day discovery. You can try the free scan at gold.safeguard.sh/scan against a branch like this to see how an old commit looks.
Quick check
What if we have no past incidents?
Pick a package that was upgraded for security reasons in your changelog, or add an old dependency version in a throwaway branch.
Is one replay enough?
It is a start. Two or three, from different ecosystems, give a fairer picture.
Last reviewed: 19 October 2026.
