Read a scanner benchmark by asking what was tested, who decided the right answers, how results were scored and who ran it. A chart can be honest and still irrelevant to your code. Treat any benchmark as a hypothesis to check on your own repositories, not as a result to quote.
Start with the test set
- What code was scanned? Synthetic test projects, popular open source repositories or real customer code behave differently. Synthetic suites often favour the tool whose authors built them.
- Does it resemble yours? Languages, build systems, age and size matter. A benchmark on small modern projects says little about a ten year old monorepo.
- Is it public? A reproducible corpus lets others check. A private one asks for trust.
Ground truth is the hard part
To count missed vulnerabilities you need to know what the right answer is. Someone had to decide. Ask who labelled the data, how, and whether the labels could favour one approach. Labels made by running one of the tested tools will reward that tool.
Scoring pitfalls
- Counting findings. More findings are not better. A high count with many false positives scores well on recall and hides the cost.
- Only recall or only precision. Both matter. A tool that flags everything has perfect recall and is useless.
- Severity filters. Results limited to critical issues can mask differences elsewhere.
- Time and effort. A tool that needs three weeks of tuning is not comparable to one that runs out of the box. Was tuning allowed, and for whom?
- Averages. A tool that is excellent in one ecosystem and absent in another can post a decent average.
Who ran it
A study by a vendor, a competitor, an independent researcher or an analyst firm each has different incentives. That does not make any of them wrong. It tells you which assumptions to test. Look for a methodology section, version numbers for every tool, dates, and a note on limitations. Absence of these is itself a signal.
Dates and versions
Tools change quickly. A result from last year may describe a product that no longer exists in that form. Check the date and version, and discount old charts.
Replace the chart with your own test
Run a small bake-off:
- Pick three to five of your repositories that differ in language and age.
- Run each candidate with default settings, then with reasonable tuning.
- Verify a random sample of findings from each, and sample what each missed using a known planted vulnerable version.
- Record time, effort and clarity of output.
The capture-recapture method shows how to estimate what all tools missed from overlap between them. Your own numbers will be small and specific, and they are the ones that apply to you.
Where Safeguard fits: the free scan gives you a result on a public repository to include in your own comparison.
Quick check
Can I trust a benchmark that includes my current vendor's tool?
Only as far as the method is open and recent. Check the corpus, labels and versions, then confirm on your own code.
Last reviewed: 19 October 2026.
