Run the same static analyzer against the same codebase twice and you expect the same output — that is the entire premise of deterministic tooling. Run an LLM-based vulnerability scanner against the same codebase twice, and the premise breaks down in ways most teams have never measured. On June 29, 2026, Snyk published VulnBench JS 1.0, a benchmark built by Liran Tal specifically to quantify that gap. The methodology was deliberately repetitive: 10 JavaScript/Express fixtures scanned under 6 different configurations, each repeated 5 times, for 300 total scan runs, checked against a reference set of 44 known findings from Snyk Code's deterministic SAST engine. The headline numbers are stark — Snyk Code held 100% F1 score with zero standard deviation across every repetition, while the best-performing LLM configuration, Claude Opus 4.6 at medium reasoning effort, landed at 75.4% F1 with a comparatively tight 0.2 percentage-point standard deviation. But averaging across runs hides the more useful finding: repeatability isn't uniform within a single model's output, and knowing which findings are stable versus which are noise is what actually determines whether you can trust a scan result enough to act on it.
What does "repeatability" actually mean for a vulnerability scanner?
Repeatability means that scanning the same code twice under the same conditions produces the same findings, and it matters because a security tool's value depends on consistency, not just accuracy on any single run. A scanner that's right 75% of the time on average but flags a different 75% each run is a fundamentally different (and less trustworthy) tool than one that reliably flags the same 75% every time — even though both could report identical aggregate metrics. Snyk Code's SAST engine is deterministic by construction: given the same source and ruleset, it walks the same control-flow and data-flow graphs and produces the same output, which is why VulnBench JS 1.0 measured it at exactly 0.0 percentage points of standard deviation across five repeated runs. LLM-based scanners introduce variability at multiple layers — sampling randomness in generation, reasoning-path differences run to run, and sensitivity to how a fixture happens to be tokenized — so the benchmark's core question was whether that variability shows up mainly as random noise or as a stable, if imperfect, signal.
How was Snyk's VulnBench JS 1.0 actually structured?
VulnBench JS 1.0 tested six configurations against ten JavaScript and Express fixtures, running each pairing five times to isolate run-to-run variance from model-to-model variance. The six configurations spanned Claude Opus 4.6 at medium and high reasoning effort, Claude Opus 4.7 Max, Claude Sonnet 4.6 at medium and high effort, and Snyk Code SAST as the deterministic control. Every one of the resulting 300 scans was scored against the same 44-finding reference set drawn from Snyk Code's own detections, then further broken into findings that matched the reference set versus "unmatched" findings the LLMs surfaced that weren't in it. That split is what let Snyk's research team separate two very different phenomena that a single aggregate F1 score would otherwise blur together: how consistently a model finds real bugs, and how consistently it also floods results with extras that may or may not be genuine.
What did the results show about consistency versus noise?
The results showed a sharp asymmetry: confirmed findings were largely stable, while extra findings were largely noise. Snyk's research team reported that 134 of 158 unique reference-matched findings — 84.8% — appeared in all five repeated runs, meaning that when an LLM correctly identified a real vulnerability from the reference set, it tended to catch it reliably rather than as a lucky one-off. Unmatched findings told the opposite story: 80 of 161 unique unmatched findings (49.7%) showed up in just one of five runs, the signature of low-confidence, non-reproducible output rather than a genuine miss by the reference set. A smaller but notable 22 unmatched findings (13.7%) appeared in all five runs despite not matching any known reference issue, which is exactly the category worth scrutinizing — persistent, reproducible findings that may represent real bugs the reference set didn't originally capture, or a systematic false-positive pattern baked into how a given model reasons about certain code shapes.
Which model actually performed best, and did more expensive mean better?
Claude Opus 4.6 at medium reasoning effort was the top performer at 75.4% F1 (68% recall, 91.5% precision) with only a 0.2 percentage-point standard deviation across runs — and notably, it was not the most expensive option tested. Claude Opus 4.7 Max scored lower on accuracy, at 68.8% F1, while consuming 5.67 times the cost per session ($0.3559 versus $0.0628) and 1.86 times the tokens (95,969 versus 51,574) of the medium-effort Opus 4.6 configuration. That inversion — higher spend correlating with worse accuracy on this specific benchmark — is a useful corrective to the assumption that scaling up reasoning effort or model tier automatically improves security-finding quality. Claude Sonnet 4.6 at medium effort came out worst on stability among unmatched findings — 61.7% of its extra, non-reference reports each showed up in only one of five runs — reinforcing that reasoning-effort settings interact with repeatability in ways that aren't obviously predictable from cost or model size alone.
Where were LLMs strong, and where did they consistently miss?
The benchmarked models were consistently strong on vulnerability classes with clear, localized patterns — command and code injection, hardcoded credentials, SQL injection, SSRF, open redirect, prototype pollution, and ReDoS — categories where the vulnerable pattern tends to sit in a small, recognizable code span. They were consistently weaker on issues that require tracking state or constraints across a wider span of code: resource-limit problems, sanitization and type-validation gaps, insecure transport configuration, and path-traversal flows that depend on chaining multiple functions together. This split lines up with what deterministic taint and data-flow analysis is built for — tracing a value's full path through a program regardless of how far apart the source and sink sit — which is precisely the class of bug SAST tools like Snyk Code are architected to catch every time, and where LLM-based review is least reliably repeatable today.
What does this mean for how security teams should combine these tools?
The practical conclusion from VulnBench JS 1.0 is not "replace SAST with LLMs" or "LLMs aren't ready" — it's that the two approaches fail in different, complementary ways, and pairing them is where the value is. Deterministic SAST gives you the guaranteed, reproducible floor: every scan of the same code returns the same 44 reference-class findings, which is exactly the property you need for a policy gate that blocks a pull request on a specific CWE class. LLM-based review adds reasoning-driven coverage on top of that floor — catching contextual issues a rule-based engine wasn't written to check for — but its output should be treated probabilistically rather than as a flat pass/fail signal, especially for the roughly half of unmatched findings that only appear in a single run out of five. A defensible workflow uses deterministic SAST as the gate and treats LLM findings as investigative leads to triage, weighting confidence by whether a finding recurs across repeated passes rather than accepting single-run output at face value.