GitHub reported that Copilot Autofix, generally available since August 2024, cuts the median time to remediate a PR-time code-scanning alert to roughly 28 minutes, down from about 1.5 hours of manual work — and in beta testing developers resolved alerts more than three times faster with it than without. That is a real, measurable win. But an academic study testing ChatGPT models as standalone smart-contract vulnerability scanners found the opposite pattern when there was no prior static-analysis pass to anchor the model's judgment: recall climbed as high as 88.2% for GPT-4, but precision cratered to just 22.6% — meaning more than three-quarters of the vulnerabilities it flagged were false alarms (arXiv:2309.05520). Both facts are true, and they describe the same underlying technology used two different ways. The pattern shows up again and again in recent research: LLMs layered on top of existing static analysis tools are becoming genuinely good at separating signal from noise, while LLMs asked to find vulnerabilities from scratch remain unreliable enough that treating them as a primary scanner is a mistake. This piece walks through where the evidence actually points — the triage wins, the detection risks, the specific mechanisms (reachability, call graphs, confidence scoring) that make the difference, and what a security team should and shouldn't trust an AI to decide unsupervised in 2026.
Where does AI measurably reduce false positives?
AI reduces false positives most reliably when it triages output from an existing static analyzer rather than generating findings on its own. A 2025 paper describing a multi-agent LLM filter called QASecClaw, evaluated against the OWASP Benchmark v1.2 (2,740 labeled Java test cases), applied its filter on top of raw Semgrep output and eliminated 88.6% of Semgrep's false positives while losing only 3.1% of true positives — pushing F1 score from 78.39% to 90.93% (arXiv:2605.01885). That matters because Semgrep's baseline on the same benchmark family runs around an 80.5% true-positive rate against a 39.1% false-positive rate — meaning nearly two out of every five flagged issues were noise before the LLM layer touched them. Datadog's security research team and a separate academic project called ZeroFalse (arXiv:2510.02534) converged on the same conclusion independently: LLMs are strong at re-judging findings a deterministic tool already produced, because the hard part — finding the candidate location — is already done. The LLM's job narrows to a much easier one: does this specific flagged line, in this specific context, actually behave the way the rule assumes it does?
Where does AI introduce new false-negative or false-positive risk?
AI introduces new risk primarily when it's trusted to find vulnerabilities from a blank slate instead of judging existing candidates. The smart-contract study cited above is a clear illustration: asking ChatGPT to scan code directly, with no prior static-analysis pass to anchor it, produced a false-positive-heavy result — more than three out of four flagged issues were noise even at GPT-4's best recall — worse on precision than a well-tuned deterministic scanner. The failure mode is asymmetric and easy to miss operationally: a false positive wastes a triage cycle, but a false negative from an LLM confidently declaring something "safe" can suppress a real finding a human reviewer would have caught by simply reading the raw scanner output. Researchers behind ZeroFalse specifically flagged this as the more dangerous direction — an LLM over-trusted to filter "noise" can quietly convert a true positive into a dismissed one, and unlike a missed rule in a deterministic scanner, there's no static ruleset to audit afterward to see what was thrown away.
What made CodeQL's 2025 changes different from an LLM bolt-on?
CodeQL's 2025 roadmap is a useful contrast because most of the gains came from deterministic engineering, not generative AI. GitHub added workflow-analysis coverage in April 2025 — new checks for missing permissions, dangerous inputs, and script injection in CI/CD workflow files — and shipped incremental analysis in May 2025 that cut PR-scan time by up to 20% by re-analyzing only changed code paths instead of the whole repository. Only in October 2025 did Copilot's coding agent start automatically running CodeQL and dependency-advisory checks against its own generated pull requests before a human ever sees them. That ordering matters: the deterministic analysis engine got faster and broader first, and the AI layer was added on top to consume that output and act on it — the same triage-not-detection pattern showing up in how GitHub actually shipped the feature, not just in academic benchmarks.
How does reachability reasoning change the false-positive calculus?
Reachability reasoning changes the calculus by giving an AI a structural fact to reason over instead of asking it to judge exploitability from code alone. Safeguard's own Griffin model, for example, takes a finding plus a call graph as input and returns a reachable / unreachable / conditional verdict with a plain-language justification at roughly 4 seconds median latency — the same "reachable" question that industry-wide reachability tooling has shown eliminates 70–85% of SCA findings that are technically present in a dependency tree but never invoked at runtime. The distinction from the blank-slate ChatGPT failure mode above is structural: Griffin isn't asked "is this code vulnerable," a semantic judgment call an LLM handles inconsistently — it's asked "does a path exist from this entry point to this line," a graph-traversal question with a checkable answer, that the model then explains in natural language. When Griffin's output is uncertain, the response includes an explicit confidence band rather than a bare verdict, which is the honest middle ground between "AI decides" and "AI explains what a deterministic system already found."
What does responsible false-positive handling actually require?
Responsible false-positive handling requires a correction path that doesn't silently retrain the model on every disagreement. Safeguard's Eagle model, which classifies packages, container layers, and model weights as benign, suspicious, or malicious with a numeric score and contributing indicators, ships a dedicated dispute workflow — safeguard malware dispute --id <finding> — with a 24-hour review SLA, and verified false positives get added to an allowlist layer that sits alongside the model rather than altering its weights. That's a meaningfully different design than a black-box classifier with no appeal path: it acknowledges the model will sometimes be wrong, gives a bounded time-to-correction, and avoids the failure mode where every override quietly degrades detection on the next case. Any team evaluating an AI-driven AppSec tool in 2026 should ask this exact question before rollout: when the model is wrong, what happens next, and how fast?
So where should a security team actually trust AI in 2026?
A security team should trust AI to triage, explain, and remediate — and should stay skeptical of AI asked to detect from a blank slate. The evidence lines up consistently across independent sources: GitHub's autofix latency numbers, the QASecClaw benchmark's 88.6% false-positive reduction, Datadog's and ZeroFalse's convergent findings on LLM-as-filter versus LLM-as-detector, and the smart-contract study's precision collapse when ChatGPT scanned from scratch all point the same direction. The practical takeaway is architectural, not philosophical: keep a deterministic scanner or call-graph engine doing the finding, let AI do the judging and the explaining on top of it, and make sure whatever AI layer you add has a visible confidence signal and a real path to correct it when it's wrong.