In June 2024, researchers Joseph Spracklen, Raveen Wijewickrama, A H M Nazmus Sakib, Anindya Maiti, Bimal Viswanath, and Murtuza Jadliwala published "We Have a Package for You!" (arXiv:2406.10279), a study accepted to USENIX Security 2025 that ran 576,000 code-generation samples through 16 large language models. The result: commercial models invented non-existent package names in roughly 5.2% of samples, and open-source models did so in roughly 21.7% — a combined 205,474 unique hallucinated package names across the dataset. Worse, 58% of those hallucinated names recurred consistently across repeated prompts and runs, meaning the hallucinations are not random noise — they are predictable enough for an attacker to pre-register the exact name an LLM will suggest, publish malware under it, and wait for a developer to run pip install or npm install. Security researchers have started calling this "slopsquatting." That is only one half of the trust problem. A separate, peer-reviewed Stanford study found that developers using an AI coding assistant not only wrote less secure code, they became more confident it was secure. This post walks through both failure modes and closes with a concrete pre-merge review checklist.
What is package hallucination, and why does it enable a new supply-chain attack?
Package hallucination happens when an LLM, asked to write code that imports a library, confidently names a package that does not exist on PyPI or npm — often because the name is plausible given the model's training distribution (a sensible-sounding combination of a known library's naming conventions) but was never actually published. The Spracklen et al. USENIX 2025 paper found this happened across 576,000 generated samples at rates of 5.2% (commercial models) and 21.7% (open-source models), producing 205,474 distinct hallucinated names. The dangerous part isn't the hallucination itself — it's the 58% repetition rate the paper documented: the same prompt, run again, often produces the same fake package name. That predictability is exactly what turns a model quirk into an attack primitive. An attacker doesn't need to guess; they can query the same open models researchers used, harvest the recurring hallucinated names, register them on public registries with malicious install scripts, and wait.
Does AI assistance actually make code less secure, or just feel that way?
Both, according to controlled research. Perry, Srivastava, Kumar, and Boneh's "Do Users Write More Insecure Code with AI Assistants?" (ACM CCS 2023) ran a user study comparing developers who used a Codex-based AI assistant against a control group writing the same tasks unassisted. The AI-assisted group produced measurably less secure code across the study's tasks. The more striking finding was on the human side: participants who used the assistant were more likely to believe their code was secure than the control group, even though it measurably wasn't. That combination — worse code, higher confidence — is what the paper frames as a false sense of security, and it's a distinct risk from hallucination: it doesn't need a malicious package to cause harm, just an LLM-authored input-handling function, a SQL query, or a crypto call that looks plausible enough that a reviewer skims past it.
Which insecure coding patterns show up most often in LLM output?
Across the Perry et al. study and follow-on replications, the recurring pattern classes are consistent: SQL statements built by string concatenation or f-string interpolation instead of parameterized queries; missing or superficial input validation (checking for empty strings but not type, length, or encoding); hardcoded credentials, API keys, or connection strings left in example code that gets copy-pasted verbatim; outdated cryptographic primitives — models frequently reach for MD5 or DES when asked to "hash a password" or "encrypt some data" simply because those calls appear constantly in the Stack Overflow and tutorial corpus the models were trained on; and insecure deserialization, such as unpickling attacker-reachable input. None of these are novel vulnerability classes — they're the same CWE categories static analysis has targeted for years (CWE-89, CWE-798, CWE-327, CWE-502). What's different is the volume: an LLM can produce them at the pace of autocomplete, across a whole file, before a human has read a single line.
Why can't code review alone catch this?
Traditional review assumes the author understood the code well enough to explain a design decision if challenged — with LLM-authored code, that assumption breaks, because neither the human committer nor the model has "intent" in the traditional sense; the model produced statistically likely tokens, not a reasoned security decision. A reviewer skimming a 40-line AI-generated function for logic correctness can easily miss that an import points to a package that has 200 GitHub stars this week and didn't exist six months ago, or that a subprocess.run() call was built with a fine cryptographic pattern for the wrong threat model. The fix isn't more review effort per line — it's shifting specific, mechanical checks (does this import exist and match the expected publisher? is this the parameterized-query form?) out of human attention and into automated gates that run on every PR, since the Stanford study's core finding is that human confidence is precisely the variable AI assistance distorts.
What should a pre-merge checklist for AI-generated code actually include?
A practical checklist has to cover both failure modes separately, because they need different tooling. For hallucination: verify every new import against the actual package registry before merge — does it exist, does it have a plausible publish history and maintainer, and is it the package you meant (not a name one edit-distance away)? Registry-level enforcement — an install-time proxy that checks typosquat distance and namespace confusion before a package ever reaches disk — closes this gap more reliably than a human eyeballing a diff, which is the exact mechanism behind Safeguard's Package Firewall, run in audit mode first to baseline before moving to block. For insecure patterns: run SAST/taint rules on every AI-authored diff specifically for CWE-89 (string-built SQL), CWE-798 (hardcoded secrets), CWE-327 (weak crypto primitives), and CWE-502 (unsafe deserialization) — the same categories the research above flags as recurring. Require that any new dependency's version be pinned and its transitive tree scanned, not just the top-level name. And explicitly do not let "the AI wrote it and it looks fine" substitute for a reviewer tracing at least one untrusted-input path through the change by hand.
How does Safeguard help close this gap?
Safeguard's Package Firewall runs as an install-time proxy in front of npm and pip, evaluating every fetch — including transitive dependencies — for typosquatting, namespace confusion, and known-malicious signatures before a package reaches a developer's machine or a CI build, with audit, warn, and block modes so teams can baseline impact before enforcing. That directly addresses the slopsquatting scenario the USENIX 2025 research describes: even if an LLM suggests a hallucinated package name and a developer runs the install command, the firewall evaluates it against real registry data at the moment of fetch, independent of whether a human reviewer caught it in the diff. On the insecure-pattern side, Safeguard's SAST and reachability engines flag the same CWE-89, CWE-798, and CWE-502 classes the Perry et al. study found AI assistants produce at elevated rates, and Griffin AI explains each finding in plain language with a suggested fix — turning the mechanical part of catching AI-generated insecurity into a gate that runs before merge, rather than a hope that a confident-looking pull request gets read carefully enough.