Treat a benchmark score as a screening input, not a verdict. Ask which test suite, which version, which rules, what false positive rate was reported alongside detection, and whether the vendor tuned the tool to the suite. Then confirm on your own repositories, where frameworks and build setups differ.
What a public benchmark is
The OWASP Benchmark is an open source test suite, a runnable application with deliberately planted vulnerabilities, designed so that static, dynamic and interactive tools can be run against it. Its page states that every vulnerability it scores is meant to be exploitable, and that the latest version is the current head of the repository rather than an old release.
That is useful. It gives vendors and buyers a shared target and a way to compare tools on identical code. OWASP's own tool selection criteria list accuracy, including false positive and false negative rates and the Benchmark score, as one input.
What it cannot tell you
A single suite covers certain languages and certain vulnerability categories. Your stack may be outside it. Your code uses frameworks, internal wrappers and build quirks that no public suite reproduces.
Scores can also drift from reality in three ways:
- Tuning to the test. A tool can be adjusted until it scores well on a known suite. The score then describes the suite, not your application.
- Selective reporting. A vendor may quote detection rate and leave out false positives, or the reverse. A tool that flags everything detects everything.
- Version mismatch. The score may describe an older engine or a different rule set from what you will run.
Questions to ask when a number is quoted
- Which suite, which version, and on what date was it run?
- What are the true positive and false positive rates, reported together?
- Which languages and vulnerability categories does the number cover?
- Was it run by you, or by an independent party? Can I reproduce it?
- Which product edition and rule set was used?
Ask for the raw output, preferably in SARIF so you can load it into your own tools. A number you cannot reproduce is marketing.
Do your own mini benchmark
Seed three or four repositories with a handful of known bug patterns from your own history, add a few safe look alikes, and run every candidate. Count what is found, what is missed, and what is flagged wrongly. This is a small sample and will not give you a rate you can quote, but it shows relative behaviour on code you care about.
Pair it with the labelling exercise from a pilot: have developers mark each finding real, low value, or not real. Compare tools using the same labels.
Reading ratings and rankings from analysts
Analyst reports and online lists can help you find candidates. They rarely test on your stack. Use them to build a shortlist, then decide with your pilot.
What to ignore
Ignore rule counts as a quality signal, ignore any claim of zero false positives, and be careful with claims of finding most vulnerabilities. OWASP's description of static analysis states that only a small percentage of application security flaws can be found automatically, and that false positives are common. A vendor claiming otherwise owes you evidence.
Quick check
Is a high benchmark score enough to buy?
No. It tells you a tool does well on one suite. Confirm on your own code, frameworks and build.
What should accompany a detection rate?
The false positive rate on the same run, plus the suite, version and rule set.
For the dependency side, Safeguard's reachability analysis and SBOM scanning are generally available, with a free check at gold.safeguard.sh/scan.
Sources
Last reviewed: 19 October 2026.
