A scanner that reports every single test case as vulnerable scores a 100% detection rate on the OWASP Benchmark. It finds every real flaw, because it finds everything.
That scanner is worthless, and the suite is built to say so — but only if you read the second number. Detection rate on its own is the most quotable figure the Benchmark produces and the one that carries the least information, which is exactly why it turns up in vendor material on its own.
What the suite actually is
The OWASP Benchmark is a deliberately synthetic Java application. Version 1.2 ships 2,740 test cases across eleven weakness categories — command injection, path traversal, SQL injection, XSS, LDAP and XPath injection, weak cryptography, weak hashing, weak randomness, insecure cookies, and trust boundary violations.
The design decision that makes it useful is that roughly half the cases are not vulnerable. Each one is the same shape as a real flaw with the dangerous part neutralised: the input is bound as a parameter instead of concatenated, or encoded before it reaches the sink, or assigned only on a branch that provably never executes.
Ground truth ships with the suite as a CSV. There is no interpretation step and no vendor in the middle. You can run it yourself.
The three numbers, and which one matters
For each category:
- True positive rate (recall) — of the genuinely vulnerable cases, the fraction the tool reported.
- False positive rate — of the deliberately safe cases, the fraction it reported anyway.
- Youden's index —
TPR − FPR. One number. 1.0 is perfect, 0.0 is a coin flip, and negative means the tool is doing worse than guessing.
Run the flag-everything scanner through that and you get TPR 1.000, FPR 1.000, Youden 0.000. The suite correctly calls it noise. Run a scanner that reports nothing and you get 0.000 across the board — also 0.000 Youden, also correctly judged.
A benchmark figure quoted without its false-positive rate is not a result. It is half of a subtraction.
Four ways the number gets misread
Recall without FPR. Covered above, and the most common. If a datasheet gives you a detection rate and no noise figure, the noise figure is the interesting one.
The pooled average hides blind spots. The suite is heavily unbalanced — SQL injection, weak randomness and XSS together are more than half the cases. A tool can post a respectable overall Youden while scoring flat zero in four categories, because the big categories carry it. Ask for the per-category table, or compute the macro average that weights all eleven equally. The gap between the two is a map of what the tool cannot see.
CWE labelling counts as a miss. The official scorer credits a finding only when the reported CWE matches the one the Benchmark assigns that category. A tool that detects a broken hash perfectly but reports it as CWE-327 (broken crypto algorithm) rather than CWE-328 (weak hash) scores zero on hashing. That is a real gap — CWE-keyed dashboards and compliance mappings break the same way — but it is a labelling gap, not a detection gap, and the two need different fixes. Any honest scorecard reports both a strict and a lenient run so you can see which you are looking at.
Synthetic is not real. The Benchmark tells you whether an engine's dataflow analysis is sound on textbook shapes. It tells you nothing about framework indirection, build systems, monorepos, or a 40-million-line codebase. A good score is evidence about the engine. It is not evidence about your repository, and a vendor who blurs the two is telling you something about themselves.
What the safe half is really testing
The most instructive failures are on cases that are not vulnerable.
A common one: the tainted value is assigned inside a branch guarded by an arithmetic condition that is constant at compile time — the sort of if ((7 * 42) - 86 > 200) construct that always resolves the same way. A scanner without constant folding sees a tainted assignment reachable and fires. A scanner with it sees dead code and stays quiet.
Nothing about that is unfair. Path-sensitivity is precisely the capability that separates a linter with a taint table from a dataflow engine, and the suite is measuring it deliberately. If a tool's false positives cluster there, you have learned something specific about its analysis rather than a vague impression that it is noisy.
Running it yourself
git clone --depth 1 https://github.com/OWASP-Benchmark/BenchmarkJava.git
# point your scanner at src/main/java/org/owasp/benchmark/testcode
# score the output against expectedresults-1.2.csv
Two things worth doing while you are there.
Score every test case, including the ones your tool said nothing about. A crashed or partial scan that quietly shrinks the denominator reports as 0/0 rather than as a wall of misses, and 0/0 looks like nothing went wrong.
Count the findings the score cannot see. A finding whose CWE does not belong to its test case's category is ignored by the scorer entirely — it moves neither TPR nor FPR. On a real codebase those are noise a developer still has to read. A tool can post a clean benchmark FPR while emitting thousands of them.
The honest use of a benchmark
Run it before you buy, on the tools you are actually considering, and read the per-category table rather than the headline. Compare Youden, not recall. Treat a zero in a category you care about as disqualifying regardless of the overall figure.
And treat any vendor number you did not generate yourself as a starting question, not an answer. The suite is public, the ground truth is public, and the whole point of a reproducible benchmark is that you do not have to take anyone's word for it.
Related reading: what SAST actually does, taint analysis versus reachability, and why reachability is the missing piece of SCA.
