Keep a small set of canary images whose findings you already know, and run your scanner against them after every update to the tool or its data. If a known finding disappears or an unexpected one appears, you catch a regression before it reaches your real results. Build the set once, keep it in version control and automate the check.
Why scanners need regression tests
Scanners change. The tool itself is updated, its parsers change, and its vulnerability data is refreshed constantly. Any of these can alter results. Usually the change is an improvement. Sometimes a parser bug drops a whole class of packages, and nobody notices because the report simply gets shorter, which looks like good news.
A canary set gives you a fixed reference. Because you know what should appear, a missing finding is visible. This is especially useful when a vendor changes something quietly, or when you move from one scanner to another and want to check that nothing important went missing.
What goes into the set
Choose images that cover your real estate and the awkward cases.
| Canary | What it checks |
|---|---|
| Image with an old package on each OS family you run | OS package parsing and distribution data |
| Image with a known vulnerable application library | Language ecosystem detection |
| Image with a package patched by a distribution revision | That fixed versions are not flagged |
| Image on an end-of-life base | That the tool warns about unsupported data |
| Multi-architecture image | That every platform is handled |
| Image with no package database, such as one built from scratch | That the tool says "nothing found to analyze", not "clean" |
| Image with a binary installed outside the package manager | The limits of the tool, so you know them |
Keep each image small and pinned by digest so it never changes under you.
Record the expected results
For each canary, write down expected findings, using stable identifiers.
A real example of a check you can build today: in the Alpine security data for the v3.21 branch, the apache2 package has fix entries, and the entry for version 2.4.51-r0 lists CVE-2021-42013. An image carrying apache2 at 2.4.50-r0 should therefore report CVE-2021-42013, since it has not reached that fix. When we queried the public OSV API for apache2 2.4.50-r0 on that branch, it returned 93 records, and the first by identifier was the record for CVE-2021-42013. Your expected file can say "must include CVE-2021-42013", and it does not need to list all 93.
Two kinds of expectation are useful.
- Must include: findings that have to appear. A missing one fails the test.
- Must not include: findings that must not appear, such as a CVE fixed by a revision. A present one fails the test.
Avoid exact totals. Counts change when data changes, and your test will fail for reasons you do not care about.
Step by step: build and run the set
- Pick six to ten canary images and pin each by digest.
- Scan each with your current scanner and review the output by hand.
- Write the must include and must not include lists, from your review and from independent checks such as the public data sources.
- Store images or their references and the expected files in version control.
- Write a script that scans each canary, extracts CVE identifiers and compares them with the expectations.
- Run it whenever the scanner version changes, on a schedule weekly to catch data changes, and before any vendor migration.
- On failure, read the difference before deciding who is wrong. Sometimes the expectation is out of date, for example because a fix was published and the finding should now disappear.
- Review the set every quarter and add a canary whenever a miss in production teaches you something.
How to handle a failure
| Failure | Likely cause | Action |
|---|---|---|
| Must include finding missing | Parser regression, data change or wrong ecosystem | Investigate before trusting reports. Raise with the vendor |
| Must not include finding present | Version comparison bug or stale data | Same |
| Scan fails to run | Tool or access change | Fix the integration |
| Result differs only in counts | Data changed | Usually ignore |
| Finding disappeared because a fix landed | Expectation stale | Update the file with a note |
Cost and effort
A set of this size scans in a few minutes and costs little to run. The larger cost is the first build, an afternoon of careful work. Most teams find it pays back the first time a vendor update changes behavior.
Questions for a vendor
- Do you publish release notes that describe parser and data changes?
- Can I pin a scanner version and update on my schedule?
- Can I pin a data snapshot for repeatable tests?
- Do you run your own regression tests, and can you share what they cover?
Common mistakes
- Using only clean images, so nothing can fail.
- Using floating tags, so the canary changes under you.
- Asserting exact counts.
- Never updating expectations, so the set rots and people stop trusting it.
Try it yourself
Click through it at your own pace, or open it full screen.
See how repeated checks over time surface a change. A canary run is the same idea applied to the scanner itself.
Where Safeguard fits
You can use the same canary approach on any scanner you evaluate, Safeguard included. Start with a free scan at gold.safeguard.sh/scan, or the CLI for scripted runs.
Quick check
How many canary images do I need?
Six to ten is enough for most teams, covering each OS family, language and awkward case you run. Add more only when production teaches you a new failure mode.
Should I assert exact finding counts?
No. Counts move whenever data changes. Assert that specific known findings are present or absent.
How often should the set run?
On every scanner version change, weekly to catch data changes, and before any migration to a different tool.
Sources
Last reviewed: 19 October 2026.
