In April 2025, researchers analyzing 16 large language models for a USENIX Security study found that when the models were asked to generate code, they recommended non-existent software packages in 19.7% of samples — producing more than 205,000 unique hallucinated package names across 576,000 code generations. That is not a quirky trivia problem about AI creativity. Attackers have already started registering those exact hallucinated names on npm and PyPI, waiting for a developer to copy-paste an LLM-suggested pip install or npm install command straight into a production build. Security researchers now call this "slopsquatting."
It is one small example of a much bigger pattern: hallucinated LLM output no longer just misleads a chatbot user — it flows into codebases, legal filings, vulnerability reports, and customer-facing systems, where it behaves like injected misinformation with real operational consequences. This post walks through how LLM hallucination becomes a concrete security risk, with dated, verifiable examples, and what engineering and security teams can do to contain it.
What Is LLM Hallucination, and Why Does It Matter for Security?
LLM hallucination is a model generating fluent, confident, and often highly specific output that is factually wrong — a package name, a legal citation, a CVE identifier, an API method — with no internal signal distinguishing it from a correct answer. That matters for security because hallucinations don't fail loudly. They fail exactly like correct answers, which means they bypass the skepticism a human would normally apply to an obvious error.
The scale of the problem is measurable and, on frontier reasoning models, has been getting worse rather than better. OpenAI's own April 2025 system card for the o3 and o4-mini models reported hallucination rates of 33% and 48% respectively on the internal PersonQA benchmark — roughly double the 16% rate of the predecessor o1 model. Vectara's public Hallucination Leaderboard, which has tracked summarization accuracy across dozens of commercial and open models since late 2023, consistently shows even top-performing models fabricating unsupported claims in a nontrivial share of outputs. When a security team treats an LLM's output as a fact rather than a hypothesis to verify, that gap becomes an exploitable trust boundary.
How Do Attackers Exploit Hallucinated Software Packages?
Attackers exploit hallucinated packages by pre-registering the exact fictitious names LLMs are statistically likely to suggest, then publishing malware under those names so the next developer who trusts the AI's suggestion pulls it straight into their build. This isn't theoretical. In 2023, security researcher Bar Lanyado ran an experiment publishing a benign decoy package under a name, "huggingface-cli," that he had observed ChatGPT hallucinating as a real dependency; it reportedly accumulated tens of thousands of downloads within months, including from large companies, before he disclosed it.
The 2024–2025 academic study referenced above (Spracklen et al., "We Have a Package for You!") formalized the pattern at scale: across 30 different prompting setups, open-source models like CodeLlama and DeepSeek hallucinated packages in roughly 21.7% of generations, while commercial models like GPT-4 series did better but still hallucinated in about 5.2% of cases. Crucially, the same hallucinated package names recurred across repeated queries 43% of the time — meaning an attacker doesn't need to guess broadly; they need to run the same popular prompts a few times, harvest the repeatable hallucinations, and squat on a small, high-yield set of names. In April 2025, Python Software Foundation developer Seth Larson gave the phenomenon its name, "slopsquatting," and flagged it as an emerging supply chain threat category distinct from traditional typosquatting.
Can Hallucinated Code Introduce Vulnerabilities Directly Into a Codebase?
Yes — beyond phantom dependencies, LLMs also hallucinate insecure patterns inside code that compiles and runs fine, which is arguably more dangerous because there's no missing-package error to catch it. A widely cited Stanford study (Perry et al., 2023) found that developers using an AI code assistant were more likely to introduce security vulnerabilities into several categories of tasks (including SQL injection and weak cryptography) than developers working without one, and — more troubling — those same developers were more likely to describe their AI-assisted code as secure. Confidence rose even as correctness fell.
Independent of hallucination research specifically, code-quality analytics firm GitClear reported in its 2024 industry study that as AI-assisted coding scaled across large codebases, the share of "copy-pasted" versus refactored code climbed sharply, an indicator that suggested output was going in with less review, not more. Combine a model that occasionally invents a plausible-but-wrong crypto call, a deprecated API, or an authorization check that doesn't actually run, with reviewers who trust the tool's fluency, and the result is vulnerable code shipped with unusually high confidence.
What Happens When LLMs Hallucinate Security Advisories or CVEs?
When LLMs are used to summarize or triage vulnerabilities, they sometimes fabricate CVE identifiers, misattribute a flaw to the wrong library version, or invent technical detail that sounds authoritative but doesn't match the actual advisory — and this has already disrupted real vulnerability-management workflows. The clearest public example comes from the curl project. Maintainer Daniel Stenberg reported in January 2025 that curl's HackerOne bug bounty program was being flooded with AI-generated vulnerability reports containing hallucinated proof-of-concepts and fabricated technical claims — reports that read as credible on first pass but described flaws that did not exist. Stenberg estimated a meaningful share of submissions as "AI slop" and said the project would ban submitters who kept using LLMs to mass-generate reports, because triaging convincing-but-false reports consumed real maintainer time that should have gone to genuine issues.
The same failure mode runs in the other direction: if a security or engineering team asks an LLM to summarize whether a dependency is affected by a given CVE, a hallucinated "not affected" answer is functionally the same as skipping the patch. Because the answer is delivered with the same fluent confidence as a correct one, it can pass through triage without a second look.
How Costly Can LLM Misinformation Become for a Business?
It has already cost real organizations real money, in disputes a court has actually settled. In February 2024, a Canadian civil resolution tribunal ordered Air Canada to honor a bereavement-fare discount that its customer-service chatbot had invented and told a customer, Jake Moffatt, that he could apply for after his flight — the airline argued the chatbot was a separate legal entity responsible for its own words, and lost. In June 2023, a federal judge in Mata v. Avianca sanctioned two New York attorneys, Steven Schwartz and Peter LoDuca, fining their firm $5,000 after they submitted a legal brief citing six court cases ChatGPT had entirely fabricated, complete with plausible-sounding case names and docket numbers. Similar attorney sanctions for AI-hallucinated citations have continued through 2024 and 2025 in multiple U.S. jurisdictions.
The market has priced hallucination risk too. In February 2023, Google's Bard chatbot made a factual error about the James Webb Space Telescope in a promotional demo; the misstep was widely reported as contributing to roughly $100 billion being wiped off Alphabet's market capitalization in a single trading day. None of these incidents involved a traditional exploit, malware, or a stolen credential. Each involved an LLM stating something false with total confidence, and a human or system downstream acting on it as if it were verified fact — which is precisely the mechanism that makes LLM misinformation a security risk rather than just a quality problem.
How Safeguard Helps
Safeguard treats LLM-originated content — suggested dependencies, generated code, AI-assisted vulnerability summaries — as untrusted input that has to earn trust through verification, the same posture we apply to any third-party artifact entering the software supply chain. Concretely, that means:
- Dependency and package verification: Safeguard checks packages pulled into a build against real registry data, known-publisher history, and package age/reputation signals, so a hallucinated or freshly squatted package name doesn't silently make it past a human who assumed the AI's suggestion was legitimate.
- SBOM-driven provenance: by maintaining an accurate software bill of materials and mapping it to verified upstream sources, Safeguard gives teams a ground truth to check AI-suggested dependencies against before they merge, rather than trusting an LLM's memory of what a real package looks like.
- Vulnerability intelligence tied to authoritative sources: Safeguard's vulnerability and CVE data is sourced and cross-checked against authoritative feeds rather than generated or summarized by an unverified model, closing the gap that let hallucinated advisories and phantom CVEs slip into other tools' triage queues.
- CI/CD gating for unverified code and dependencies: policy checks in the pipeline can block builds that introduce unrecognized or unverifiable packages, giving teams a technical backstop for the moment a developer trusts an AI suggestion a little too quickly.
- Audit trail and review evidence: for teams operating under SOC 2 or similar compliance regimes, Safeguard preserves the change and approval history needed to show that AI-assisted contributions, like any other code, went through verification before shipping.
LLMs are not going away from the software development lifecycle, and they shouldn't have to. But their fluency is exactly what makes hallucinated output dangerous — it looks like every other correct answer they give. Treating AI-generated packages, code, and security summaries as claims to verify, not facts to trust, is the practical difference between an isolated model quirk and a supply chain incident.