In March 2025, researchers published the CASTLE benchmark (arXiv 2503.09433), running 13 traditional static analyzers, 2 formal-verification tools, and 10 large language models against 250 hand-crafted C programs spanning 25 CWE categories. The result was not a clean win for either side: static analyzers buried real findings under high false-positive rates that demanded heavy manual triage, while LLM performance was, in the researchers' words, "strong but unstable" — accurate on some vulnerability classes and close to guessing on others. That split matters because the two approaches work in fundamentally different ways. Symbolic-reasoning tools like CodeQL and Semgrep build an abstract syntax tree and a call graph, then trace tainted data from a declared source to a declared sink using fixed inference rules — deterministic, reproducible, and auditable. LLM-based scanners instead pattern-match over code semantics learned during training, producing probabilistic output that can change between runs and rarely comes with a formally verifiable trace. This piece breaks down what that distinction actually costs and buys you, using published benchmark data rather than vendor claims, and where a defensive AppSec program should draw the line between the two.
What's the actual technical difference between symbolic and LLM-based scanning?
Symbolic-reasoning SAST tools parse source into an AST, build a call graph across functions and files, and apply fixed dataflow rules to determine whether untrusted input from a declared source — a request parameter, a CLI argument — can reach a declared sink, like a SQL query or a shell exec, without passing through a sanitizer. Every finding carries an explicit, replayable hop-by-hop trace, and running the same code through the same ruleset twice produces the same result. LLM-based scanners instead feed code (or diffs) to a trained model and ask it to judge whether a pattern looks like a known vulnerability class. There's no fixed inference rule being applied — the model's internal reasoning isn't a formal proof, and its natural-language explanation is generated after the fact, not derived from a traceable dataflow graph. That's the core tradeoff: symbolic tools give you provable coverage of the rules they encode; LLMs give you fuzzier, broader semantic pattern-matching with no guarantee the same input produces the same output twice.
How much noise does symbolic analysis produce on its own?
Semgrep's own documentation is explicit that the tool "has no soundness guarantees" — it makes "reasonable," not worst-case-sound, assumptions, which means it can both miss real bugs and flag non-issues by design, including documented gaps like eval-driven state changes and taint loss across HashMap keys and values. That's not a defect unique to Semgrep; it's an inherent property of dataflow analysis that has to balance completeness against runtime and noise. A study published via escholarship.org measured this directly: Semgrep alone achieved an 80.49% true-positive rate against its benchmark, but also a 39.09% false-positive rate — meaning roughly two in five flagged issues required a human to confirm they weren't real before any remediation work could start. At enterprise scale, with thousands of findings per scan, that false-positive rate is what drives alert fatigue and causes teams to quietly disable rules rather than triage them.
Can LLMs actually find vulnerabilities symbolic tools miss?
Sometimes, but with a real cost in reliability. IRIS, a GPT-4-based vulnerability-detection system presented at ICLR 2025, reported an 84.82% false discovery rate on its evaluation set — high in absolute terms, but 5.21 percentage points lower than CodeQL's baseline false discovery rate on the same benchmark, suggesting LLM-assisted analysis can meaningfully cut noise relative to a pure symbolic baseline without eliminating it. The same escholarship.org study found that layering an LLM (o1-mini) review pass on top of Semgrep's raw output improved precision further than either tool alone. The pattern across both results is consistent: LLMs are useful as a second-pass filter that reasons about context a fixed dataflow rule can't — is this sanitizer actually effective here, is this code path realistically attacker-reachable — but neither result shows an LLM finding a class of bug that a well-built symbolic engine structurally cannot detect once instrumented for it.
Why does explainability matter for compliance, not just triage?
A dataflow trace from a symbolic tool is evidence: an auditor, a compliance reviewer, or a developer can follow the exact ordered hops from source to sink and independently verify the finding is real, which is precisely the kind of reproducible artifact SOC 2 and similar audits expect for control evidence. An LLM's explanation of why it flagged something is a generated narrative, not a derivation — it can sound confident and still be post-hoc rationalization for a pattern-match that doesn't hold up on inspection, and because output can vary run-to-run, the same code scanned twice isn't guaranteed to produce the same justification. That doesn't make LLM explanations worthless — plain-language rationale is genuinely useful for a developer deciding how to fix something — but it means LLM output functions best as an explanatory layer on top of deterministic evidence, not as the evidence itself when someone has to sign off on a control.
Can combining the two approaches reduce false positives without losing evidence?
The published evidence points toward yes, when the LLM sits downstream of a deterministic engine rather than replacing it. Semgrep has claimed up to a 98% reduction in high/critical false positives from added cross-file and cross-function dataflow analysis — a vendor figure worth treating skeptically until independently reproduced, but directionally consistent with the escholarship.org result showing Semgrep-plus-LLM-review outperforming Semgrep alone on precision. The practical architecture this points to: run the symbolic engine first to get a reproducible, auditable set of source-to-sink paths, then use an LLM to triage and explain — not to originate — findings, flagging which ones a human should look at first and why, while leaving the underlying dataflow trace untouched as the evidence of record.
How Safeguard Helps
Safeguard's SAST engine is built on the symbolic side of this comparison by design: it traces untrusted input from declared sources to sinks across JavaScript/TypeScript, Python, and Java, and every finding carries an explicit, ordered dataflow trace along with a CWE and OWASP mapping — the reproducible evidence trail that compliance workflows and PR reviewers both need. Griffin, Safeguard's AI layer, reads that same dataflow trace and reachability verdict to explain each finding in plain language and draft an auto-fix pull request, so the deterministic trace stays the artifact of record while the LLM does the job it's actually good at — narrating and prioritizing rather than originating a finding from scratch. Reachability analysis narrows the queue further before anything reaches a developer, filtering unreachable paths from the urgent backlog using the same call-graph reasoning that produced the original finding, so AI-generated explanations are always attached to a symbolic trace someone can independently verify.