GitHub's Copilot Autofix went generally available in August 2024, and by GitHub's own account it now covers more than 90% of alert types across JavaScript, TypeScript, Java, and Python, producing usable suggestions for roughly two-thirds of found vulnerabilities "with little or no editing." A 2025 expansion added coverage for another 29% of CodeQL alert types, a 270% jump in autofix availability for that group. GitHub's beta data also showed vulnerabilities with a fix suggestion attached get remediated 3x faster on average — 7x faster for XSS, 12x faster for SQL injection. Those numbers describe throughput, not correctness, and that gap is the subject of this post. A 2026 academic study that graded 319 LLM-generated patches against 64 real Java CVEs in the Vul4J benchmark found only 24.8% were both secure and functional; 51.4% failed on both counts. A separate large-scale study of over 20,000 GitHub issues found LLM and agentic auto-repair patches introduced new vulnerabilities at roughly 9x the rate of human-written ones. Autofix tooling is now fast enough to change your MTTR chart. Whether it's safe to merge unreviewed is a separate question, and this post lays out how to answer it.
Why do AI-generated patches fix the finding but not the vulnerability?
Because most autofix tools optimize for making the flagged line stop matching a pattern, not for closing the actual security-relevant behavior the pattern was a proxy for. The 2026 "Why LLMs Fail" study (arXiv 2603.10072) measured this directly: across its 319 generated patches, the mean functional-preservation score was 0.832 — models are quite good at not breaking the build — but the mean vulnerability-fix score was only 0.251. That asymmetry is the tell. An LLM asked to fix a SQL injection will often wrap the existing string-concatenated query in a try/except, add input length checks, or rename a variable — changes that look responsive to a linter's complaint but leave the untrusted data flowing into the query unchanged. The model has strong signal for "the code still runs" (it can often execute the test suite mentally or literally) and weak signal for "the exploit path is closed," because that requires reasoning about attacker-controlled data flow the way a taint analyzer does, not the way a next-token predictor does.
How much worse is an AI patch than a human one at introducing new bugs?
According to a 2025 large-scale study of automated program repair (arXiv 2507.02976, "How Safe Are AI-Generated Patches?"), which analyzed LLM and agentic repair patches across more than 20,000 real GitHub issues drawn from SWE-bench-style benchmarks, AI-generated patches introduced new vulnerabilities at approximately 9 times the rate of human-authored patches for the same issues. This isn't a claim that AI patches are usually malicious or reckless — it's that patch generation is a narrow optimization (make the failing check pass) and models don't automatically reason about the blast radius of a change the way an experienced reviewer scans a diff for "what else does this touch." A one-line dependency bump can silently change a transitive package's default TLS behavior; a "sanitize this input" patch can introduce a second, differently-shaped injection point three lines below the one it fixed. The base rate of human patch defects isn't zero, but a 9x multiplier on a security-relevant change is not a rounding error — it's a reason no autofix should land without an independent check.
What does a validation-first pipeline actually add before merge?
A validation gate that combines automated testing with a second, independent analysis pass measurably raises correct-patch rates over an LLM's first-draft output. The VRpilot study (arXiv 2405.15690, 2024) tested exactly this: adding an automated reasoning and patch-validation feedback loop — where a generated patch is checked and the model is given feedback to retry before a human ever sees it — improved correct-patch rates by 14 percentage points for C and 7.6 for Java over an unvalidated LLM-repair baseline. The mechanism is unglamorous but effective: run the existing test suite against the patched code, then re-run the same static/taint analysis that originally flagged the vulnerability against the new version, and only surface a patch as "ready" if both come back clean. Neither check is sufficient alone — tests catch functional regressions but rarely exercise the exact exploit path, and re-scanning catches the vulnerability class but won't tell you a patch broke an unrelated caller. Run together, they catch most of what a single pass misses.
What should a human reviewer specifically check in an AI-generated diff?
A reviewer should treat an AI patch diff differently from a human PR: read it backwards from the exploit, not forward from the code. Start by re-confirming where the untrusted input actually enters the function the patch touches — if the fix adds validation somewhere that isn't on that path, it's cosmetic. Second, check whether the fix changes behavior (upgrades a dependency major version, changes a default) versus just adding a guard clause; breaking-change risk and security-correctness risk require different scrutiny and neither substitutes for the other. Third, look for the specific failure pattern the "Why LLMs Fail" study flagged as most common: exception handling or logging added around a dangerous call instead of removing the danger, which suppresses the symptom (a crash or an alert) while leaving the vulnerability live. Finally, confirm the patch doesn't only address the single reported instance when the same insecure pattern appears elsewhere in the file — LLMs are notably prone to fixing the exact line cited in the finding and leaving an identical sibling untouched two functions down.
How does Safeguard's approach to AI remediation build in these guardrails?
Safeguard's AI Remediate capability, powered by Griffin AI, is built around the same sequence this post argues for rather than a one-shot "generate and merge" flow. Griffin AI generates the fix, then runs an impact assessment that classifies the change into Safe, Minor, Major, or Critical breaking-change tiers before a pull request ever opens — so a dependency bump that changes a public API surface gets flagged differently than a config default that doesn't. Test Integration is a configurable gate that runs the existing test suite against the patched branch before the PR is created, and Require Approval keeps a human decision point in front of merge rather than auto-merging on green tests alone. None of this replaces re-running the original scanner against the patched code as a second, independent check of the vulnerability itself rather than just the test suite — that step, paired with reachability context so a reviewer knows whether the flagged path is actually attacker-reachable, is what closes the gap the Vul4J and SWE-bench studies both point to: fast isn't the same as fixed.