In August 2021, researchers from NYU published a study that should have slowed the industry's rush to ship AI-generated code — it didn't. "Asleep at the Keyboard?" (Pearce et al., IEEE S&P 2022, arXiv:2108.09293) ran GitHub Copilot against 89 scenarios mapped to MITRE's CWE Top-25 and collected 1,689 completions. The result: 39.33% of the top-ranked suggestions, and 40.73% of all suggestions across the sample, contained a security vulnerability — SQL injection, buffer overflows, and other CWE-listed patterns baked in by default. That study is now almost four years old, and the risk surface has only gotten more specific. A comprehensive USENIX Security 2025 study (Spracklen et al., first posted in 2024) generated 576,000 code samples across 16 LLMs and found that 19.7% of recommended packages didn't exist at all — hallucinated names that attackers can pre-register, a technique now widely known as "slopsquatting." Add a third, quieter problem — license contamination with no SPDX trail — and AI-generated code turns out to fail in ways that look nothing like a human developer's mistakes. This post walks through what the data actually says, and what a review guardrail needs to catch.
How often does AI-generated code actually contain vulnerabilities?
Close to 40% of the time, according to the most-cited controlled study on the question. Pearce et al.'s NYU/IEEE S&P 2022 paper is the baseline the field still refers back to: 1,689 Copilot completions, 89 scenarios drawn from MITRE's CWE Top-25, and a 39.33% vulnerability rate in the single best-ranked suggestion per scenario (40.73% across all suggestions). The vulnerable completions weren't exotic — they were the textbook insecure defaults: string-concatenated SQL queries, missing bounds checks, weak randomness for security-sensitive values. What makes this durable rather than a 2021-era artifact is the mechanism, not the specific model: an LLM trained on public code reproduces the statistical average of how that pattern is usually written, and a large fraction of public code was never written with security as the first constraint. Newer, larger models have improved on narrow benchmarks, but the underlying training-distribution problem — completions reflect what's common, not what's safe — hasn't gone away, which is why treating AI output as a first draft, not a merge-ready diff, remains the operative discipline.
What is "slopsquatting" and how big is the hallucinated-package problem?
Slopsquatting is the practice of registering a package name that an LLM hallucinates, so that the next developer who copy-pastes AI-suggested install commands pulls down attacker-controlled code instead of a 404. The term has enough traction to have its own Wikipedia entry. The underlying numbers come from a single comprehensive study by Spracklen et al. (posted to arXiv in 2024, published at USENIX Security 2025) spanning 576,000 code samples generated by 16 different LLMs: 19.7% of recommended packages were hallucinated outright, comprising over 205,000 unique nonexistent package names, with open-source models hallucinating at 21.7% versus 5.2% for commercial models. The same study found the problem was durable rather than random noise — when researchers reran hallucination-triggering prompts ten times each, 43% of hallucinated package names recurred every single time, and roughly 38% showed moderate string similarity to real package names, which is exactly the setup slopsquatting needs: a predictable, plausible-looking name worth squatting on in advance. A 2026 re-evaluation on newer frontier models (Claude Sonnet 4.6, Claude Haiku 4.5, GPT-5.4-mini, Gemini 2.5 Pro, and DeepSeek V3.2) found the rate had compressed to roughly 4.6–6.1% — real progress, but not zero, and still enough to matter at CI scale across thousands of daily pip install or npm install suggestions.
Why do commercial and open-source models hallucinate packages at such different rates?
The Spracklen et al. study's 21.7%-versus-5.2% gap between open-source and commercial models points to training and RLHF investment, not architecture. Commercial labs run heavier post-training alignment passes that penalize confidently-wrong outputs, and they've had more incentive to fix a failure mode that generates support tickets and bad press. Open-source models, often fine-tuned by smaller teams on narrower code corpora, are more likely to extrapolate a plausible-sounding package name — pandas-utils, react-fetch-helper — that fits the naming convention of an ecosystem without checking it exists. Neither category hits zero, though: the 2026 re-evaluation's 4.6–6.1% floor across frontier models suggests hallucination is a persistent property of next-token prediction over a package namespace, not a bug that scales away with model size alone. That's the practical takeaway for a review process — model choice changes the odds, but doesn't remove the need for a runtime existence check on every AI-suggested dependency before it reaches a lockfile.
Can AI-generated code introduce license risk without anyone noticing?
Yes, and it's arguably the least-monitored of the three risk classes because it produces no crash, no failed test, and no CVE. LLMs trained on public code — including GPL and AGPL-licensed repositories — can reproduce recognizable snippets or whole functions without carrying any of the license or attribution metadata that governed the original. A developer who accepts a suggested block of code has no SPDX header, no NOTICE entry, and often no way to know the snippet's provenance at all; the license obligation simply doesn't travel with the text. This is a distinct failure mode from hallucinated dependencies — there's no missing package to trip a build, no typosquat to register. It's flagged as an open concern across GitHub's own Copilot documentation and repeated legal commentary since Copilot's 2021 launch, precisely because it's structurally invisible until an audit, an acquisition due-diligence pass, or a competitor's legal team goes looking. Treating every AI-authored function as license-unknown until classified is the only way to close that gap.
What does a real guardrail against these three risks look like?
A guardrail has to intercept AI-authored code as untrusted input at the same gate any third-party contribution passes through — not as a special, faster lane. Against hallucinated packages, that means a dependency existence-check at PR time before install: does this package name actually resolve on the registry, and if it does, is it owned by a publisher matching the expected identity, not just a name that looks close enough? Safeguard's guardrail catalog documents a built-in rule for exactly this adjacent case — "Typosquat detected: block any package whose name is within edit-distance 2 of a top-1000 package but not owned by the expected publisher" — which catches the slopsquatting pattern where an attacker pre-registers a plausible AI-hallucinated name. Against license contamination, Safeguard's license-compliance policy engine classifies every new direct dependency's license against an allow/deny list and raises a blocking check at PR time when a denied or unreviewed license shows up, with the same enforcement point extending through CI and admission. Against insecure defaults, the fix is unglamorous but proven: run the same SAST/CWE-mapped rules against AI-generated diffs that you'd run against a human's, because Pearce et al.'s 40% figure says the defaults can't be trusted at face value.