Snyk published a new research benchmark this June — VulnBench JS 1.0 — that ran 300 repeated vulnerability-finding scans against 10 JavaScript fixture projects to answer a question the industry has been dancing around since agentic security review took off: if you point the same LLM at the same code with the same prompt, does it find the same bugs twice? The answer, backed by five repetitions across six configurations, is a qualified no — and the shape of that "no" has real implications for anyone deciding how much to trust an AI agent's security output.
Inside the Experiment
VulnBench JS 1.0 is deliberately narrow and deliberately rigorous. Snyk built 10 small-to-medium Express.js fixture applications — ranging from single-file snippets to a fuller todo app with server routes, database state, file uploads, and frontend JavaScript — and seeded them with 44 known reference findings drawn from Snyk Code's own static analysis engine. Each of the 10 tasks was run under 6 configurations, and each configuration was repeated 5 times, for a total of 300 identical runs. The primary agent under test used the Claude Code harness built on Anthropic's TypeScript Agent SDK, giving the model full agentic latitude — reading files, tracing call graphs, reasoning about data flow — rather than a single-shot classification prompt.
That design choice matters. Most prior "can an LLM find vulnerabilities" research reports a single accuracy number against a static test set. VulnBench JS 1.0 instead asks the harder, more operationally relevant question: given identical inputs, identical harness, and identical model, how consistent is the output across runs? Consistency, not just accuracy, is what determines whether a finding can be trusted enough to block a merge, file a ticket, or route to a human reviewer without re-verification every time.
A Sharp Split: Two Very Different Populations of Findings
The headline result is a bifurcation. Snyk separated the LLM's output into two buckets — findings that matched a known Snyk Code reference vulnerability, and everything else the model flagged that didn't match a reference.
The reference-matched findings behaved almost like deterministic output. Across the 158 unique reference-matched findings the model produced, 134 of them — roughly 85% — showed up in all five repetitions of their respective run. When the model was looking at a real, previously-cataloged vulnerability, it found it reliably, run after run.
The unmatched findings told the opposite story. Across the 250 model runs that produced extra, non-reference findings, the model surfaced 161 unique unmatched findings in total. Of those, only 22 appeared in all five repetitions — while 80, roughly half, showed up in exactly one of the five identical runs and never again. In other words: about half of everything an LLM security reviewer flags outside of known, well-characterized vulnerability patterns is a one-time event. Ask the same model the same question five minutes later, and it may never mention it again.
Why the Split Exists
The pattern lines up with how these models actually work. Reference-matched findings tend to correspond to well-known, structurally clean vulnerability classes — SQL injection via string concatenation, command injection through unsanitized exec calls, path traversal on unchecked file parameters — the kind of pattern-complete bug class that both a static analyzer and a language model have seen thousands of times in training and reference data. The signal is strong and the model converges on it reliably, because there's very little ambiguity in what "wrong" looks like.
Unmatched findings live in a much fuzzier space. They include speculative "this could theoretically be an issue if X" reasoning, chains of hypothetical exploitation the model constructs during agentic exploration, and genuinely novel observations that a reference-based static tool wouldn't catch by design (since it has no reference to match against). Some of these are real findings SAST tools miss. Many are the model reasoning itself into a plausible-sounding but ultimately non-reproducible conclusion — an artifact of sampling variance, exploration path, and how much of the codebase the agent happened to traverse before generating a report on a given run.
Snyk's own framing is notably restrained: the paper does not claim LLMs are unreliable at security review, nor that SAST is obsolete. It concludes that Snyk Code SAST remains deterministic and systematically better at exhaustively enumerating known data-flow sink patterns, while agentic LLM review adds a complementary layer of contextual, exploratory judgment — but one whose extra findings need independent verification before they're actionable, because you cannot assume a single run gives you the full — or even a stable — picture.
The Broader Signal for the Industry
VulnBench JS 1.0 lands at a moment when nearly every security vendor is racing to ship an "AI-powered vulnerability discovery" feature, and when engineering leaders are under pressure to adopt agentic code review as a substitute for scarce security headcount. This benchmark is one of the first rigorous, publicly documented attempts to quantify the repeatability tax on that promise, rather than just its raw hit rate.
The practical takeaway for any team evaluating LLM-based security tooling — whether built in-house on top of a foundation model API or bought as a packaged product — is that a single scan's findings list is not the ground truth. A vulnerability an agent reports once, in one run, carries meaningfully less evidentiary weight than one it reports across repeated, independent passes, and dramatically less weight than one confirmed by a deterministic analysis engine. Any workflow that auto-files tickets, auto-blocks builds, or auto-generates fixes directly from raw LLM output — without a corroboration step — is implicitly betting that the unmatched-finding noise rate Snyk measured (roughly 50% one-off, single-run findings) doesn't apply to its own pipeline. VulnBench JS 1.0 is a data point suggesting that bet is currently unfavorable, at least for JavaScript codebases evaluated with today's agentic harnesses.
This doesn't argue against using LLMs for security review — it argues for architecting around their statistical nature the same way any engineering discipline architects around a noisy sensor: through repetition, cross-validation against a deterministic source of truth, and reachability context that lets a human or downstream system triage signal from speculation quickly.
How Safeguard Helps
Safeguard is built around exactly this insight: AI-generated findings are only as valuable as the corroboration layered underneath them. Griffin AI, our security research agent, is paired with deterministic reachability analysis so that every AI-surfaced finding is checked against whether the vulnerable code path is actually exploitable in your running application — filtering out the kind of one-off, unreproducible noise VulnBench JS 1.0 documented, before it ever reaches an engineer's queue. Our SBOM generation and ingestion pipeline gives that reachability engine a precise, version-accurate map of every dependency in your stack, so matches against known CVEs are grounded in your real software composition rather than a single model's unverified guess. When a finding clears both the reachability bar and repeatable-signal threshold, Safeguard can generate an auto-fix pull request directly, turning a verified vulnerability into a reviewable code change instead of another ticket that needs manual triage. The result is a pipeline that gets the exploratory value of agentic AI review without inheriting its repeatability problem — because determinism, not a single lucky scan, is what should gate what ships.