Ask a static analysis vendor how accurate their engine is and you will get a recall figure, a customer quote, or a change of subject. Ask them for a score on a public test suite with labelled ground truth and the conversation gets shorter.
The OWASP Benchmark exists to end that conversation. It is a deliberately synthetic Java application containing thousands of test cases, each labelled as a genuine vulnerability or a decoy that looks like one. Run your engine over it, compare the findings to the answer key, and you have a number you did not choose.
The metric that matters
The Benchmark scores on the Youden index:
J = TPR - FPR
True positive rate minus false positive rate. A perfect tool scores 1.0. A tool that flags nothing scores 0.0. A tool that flags everything also scores 0.0, because its TPR of 1.0 is exactly cancelled by an FPR of 1.0.
That cancellation is the entire point, and it is why Youden rather than recall. Recall alone is trivially gameable: report every line as vulnerable and you have perfect recall and a useless product. Precision alone is gameable in the other direction: report only the one pattern you are certain about and enjoy your 100%. Youden punishes both strategies, which makes it the honest single number for a detector whose errors run in both directions.
Anything meaningfully above zero beats a coin flip. Published results for mature tools sit across a wide band, and the interesting thing about the band is how wide it is — the gap between "we run some regexes" and "we build a call graph and track taint" shows up immediately.
Wiring it up
The mechanics are less work than they look:
- Get the suite. Clone the Benchmark repository and build the test application. It is a Maven project; the test cases are generated Java source under a predictable path.
- Scan it. Point your engine at the source tree and collect findings. You want the file, the line, and the rule that fired.
- Map your rules to CWEs. The answer key is keyed by CWE. If your rule IDs are not already CWE-tagged, this mapping is the real work, and doing it forces a useful audit of what your rules actually claim to detect.
- Compare. For each test case, did you report a finding of the expected CWE? Count TP, FP, TN, FN, compute per-category and overall.
Build it as a command in your own toolchain rather than a one-off script. The value is in re-running it on every rule change; a benchmark you ran once is a marketing exercise, a benchmark in CI is a regression test.
Read the per-category numbers, not the headline
The aggregate hides everything useful. Break the score down by CWE category and the picture is almost always uneven:
- Injection categories — SQL, command, LDAP — tend to score well. The taint pattern is clear, the sinks are enumerable, and the decoys are distinguishable by dataflow.
- Cryptography categories — weak hashing, weak randomness, insecure ciphers — score well for a boring reason: they are essentially API-presence checks.
- Path traversal and trust boundary categories are where engines diverge, because the decoys are specifically built to defeat pattern matching and require actually following the data.
- XSS is often the worst, because the sanitiser question ("was this escaped correctly for this context?") is genuinely hard.
A high aggregate built on the cryptography categories tells you the engine can grep. The dataflow categories tell you whether it can analyse.
The result you have to be willing to act on
Here is the part nobody writes up.
If your measured Youden index comes back at, say, 0.325 — better than chance, well short of good — you now have a number that contradicts your marketing page. The whole value of measuring is that the answer is allowed to be inconvenient, and the correct response to a mediocre score is to not publish the claim it fails to support.
That is uncomfortable, and it is also the only reason the exercise is worth doing. A benchmark you would only cite if it flattered you is not a measurement; it is a search for a quotable number.
Publish it internally, set a threshold below which accuracy claims are off-limits, and treat improving the score as engineering work with a scoreboard rather than a vibe.
The side effect: it will break your engine
Running a few thousand adversarially-shaped Java files through a parser tends to find things that a curated test corpus never will.
Benchmark cases are generated from templates and deliberately include structures a hand-written test suite would not: deeply nested control flow, unusual but legal syntax, generated code with no formatting conventions, files that exist only to look like other files. That is a fuzzing corpus wearing a benchmark's clothes.
In our case the first full run surfaced a fatal crash in the engine — on input that a real customer repository could plausibly contain, and that our own fixtures never produced. That crash was worth more than the score.
Which is the practical argument for doing this even if you are confident about your accuracy: the benchmark is the largest, weirdest, most adversarial pile of source your engine has ever been asked to parse, and it comes with an answer key. Both halves are useful.
Its limits, stated plainly
The Benchmark is synthetic Java. It does not tell you how your engine behaves on a real Spring application with sixty dependencies, on Python, on TypeScript, or on code that spreads a taint path across four files and an ORM.
So it is a floor, not a certificate. A tool that scores badly here will not do better on harder input. A tool that scores well here has cleared the easiest fair test that exists — which is a claim worth making precisely because it is a modest one.
