Score each scanner on the same images, with the same criteria, and weight the criteria before you see results. Five areas cover most of what matters: coverage, accuracy, fix guidance, workflow fit and evidence. Give each a pass condition you can test, so the decision does not rest on a demo.
Set the rules before the trial
Agree on three things first: the test images, the weights and the pass conditions. If you write them after seeing results, the scores drift toward whichever tool looked strongest. Use the image inventory idea in how to build a container image inventory to pick a sample that covers each base OS family and language you run.
NIST's Application Container Security Guide, SP 800-190, explains the security concerns that come with containers and recommends ways to address them. It is a reasonable reference to read alongside your own risk list when deciding what to test.
The scorecard
| Area | What to test | Pass condition (example) | Weight |
|---|---|---|---|
| Coverage | OS packages, language packages, base images, multi-arch, images with no shell | Every image in the sample is scanned and package lists match your own spot checks | High |
| Accuracy | Known vulnerable packages, known patched packages, backported fixes | Planted vulnerable packages found; patched ones not flagged | High |
| Fix guidance | Fixed version shown, base image advice, unfixed items labeled | Each finding says what to change and where | Medium |
| Workflow fit | CI step, registry scan, exit codes, ticket links, read-only access | Runs in your pipeline with a read-only credential | Medium |
| Evidence | Dated results, digest recorded, export formats, history | You can show an auditor what was scanned and when | Medium |
Weights are yours to set. A team with a strong platform group may weight fix guidance lower than a team with none.
How to run each test
For coverage, take your sample and compare the scanner's package list with the image's own package manager output. Note anything missing.
For accuracy, plant a package you know is vulnerable and one you know was patched in a later distribution revision. Check both. Include a finding that depends on a backported fix, since these are a common source of wrong results.
For fix guidance, check that the tool separates fixable from unfixed findings and tells you whether the fix is in the base image or in your layers.
For workflow, run the scan in a real pipeline and look at how long it takes, what exit codes it returns and how a failure appears to a developer.
For evidence, export a result and read it as an auditor would. Does it name the digest, the date and the data used?
Score and decide
Score each area from zero to five, multiply by weight, and add up. Then do two checks the numbers cannot do. Ask each vendor to explain the three findings it got wrong in the trial, and how it would handle them. Honest answers tell you more than a top score.
Also record cost of ownership questions as notes, not scores: who maintains the integration, how upgrades work and what happens if the vendor service is down.
A worked example: scoring two scanners on one afternoon of tests
A team tests two unnamed scanners on a sample of eight images using the scorecard above. They set weights before starting: coverage 30, accuracy 30, fix guidance 15, workflow 15, evidence 10. The scores below are invented to show the arithmetic.
| Area | Weight | Scanner A (0 to 5) | Scanner B (0 to 5) | A weighted | B weighted |
|---|---|---|---|---|---|
| Coverage | 30 | 5 | 4 | 30 | 24 |
| Accuracy | 30 | 3 | 5 | 18 | 30 |
| Fix guidance | 15 | 4 | 3 | 12 | 9 |
| Workflow fit | 15 | 4 | 4 | 12 | 12 |
| Evidence | 10 | 3 | 4 | 6 | 8 |
| Total | 100 | 78 | 83 |
Weighted score is each score divided by five, multiplied by the weight. Scanner B wins by five points, mostly on accuracy. But the team notices Scanner B missed two images with no shell, a coverage failure that the score hides. So they add a rule: any image family that a scanner cannot scan is a gap to resolve before purchase, regardless of total.
The lesson is that a score ranks tools but does not replace reading the failures.
Step by step: run the evaluation
- Choose eight to twelve images covering every base family and language.
- Plant two known vulnerable packages and one known patched package in test images.
- Fix weights and pass conditions in writing.
- Give every vendor the same instructions and the same images.
- Collect raw outputs, not summaries.
- Score each area independently, ideally by two people who then compare.
- List every gap and failure, and ask each vendor about it.
- Decide, then record why, so you can revisit later.
Test cases that reveal differences
- An image with no package manager data, such as one built from scratch.
- A multi-architecture image.
- An image whose base is past end of support.
- An image with a language runtime and application libraries.
- An image with a vulnerability that the distribution fixed by backport.
- A very large image, to see time and memory behavior.
Questions on evidence and ownership
Ask who owns the integration after purchase, how upgrades are tested, and what happens when the vendor's service is unavailable during a build. These do not belong in the score, but they belong in the decision.
Common mistakes
- Choosing weights after seeing results.
- Testing on clean images only, so nothing differentiates the tools.
- Reading vendor benchmarks instead of running your own images.
- Skipping the failure review and trusting the total.
Try it yourself
Click through it at your own pace, or open it full screen.
Use it to see what a finding looks like, then check whether each scorecard row has a matching piece of evidence.
Where Safeguard fits
You can run Safeguard through this same scorecard. Start with the free scan at gold.safeguard.sh/scan, or the CLI for pipeline tests.
Quick check
How many images should a trial include?
Enough to cover each base OS family and language you run, plus a few images you know are vulnerable. A small, deliberate sample beats a large random one.
Should the scorecard include price?
Keep it separate. Score capability first, then compare cost for the options that pass.
Should two people score independently?
Yes. Independent scores that are then compared catch bias and reveal criteria that are unclear.
Sources
Last reviewed: 19 October 2026.
