A USENIX Security 2025 paper tested 16 code generation models on 576,000 code samples and found invented package names in every group: at least 5.2% on average for commercial models and 21.7% on average for open-source models. The paper calls the problem persistent and systemic. Here is how to read those figures.
What the paper studied
The authors looked at package hallucinations, which they define as fact-conflicting errors where a code generating model recommends a package that does not exist. They treat this as a new kind of package confusion attack on the software supply chain: if an attacker registers a name that models keep inventing, a developer who installs it without checking may run the attacker's code.
The abstract gives these figures:
- 16 popular code generation models
- 576,000 code samples
- two programming languages
- two prompt datasets
- 205,474 unique hallucinated package names
Reading the percentages
The 5.2% and 21.7% figures are averages across the models in each group, under the paper's test conditions. They are not a rate you will see in your own editor. Your result depends on the model, the language, the prompt style and whether the tool checks suggestions against a registry.
Two cautions:
- Do not quote them as "AI writes fake packages X% of the time." The paper reports a rate for its samples. Prompts in real work differ.
- Do not dismiss them either. Even a low rate adds up when many developers accept many suggestions per day.
The abstract also says mitigation strategies significantly reduced hallucinations while maintaining code quality. It does not say they remove the problem, and this post does not claim which mitigation worked best. Read the paper if you need that detail.
The number that matters most
The count of unique names, 205,474, says something different from the rate. It means the space of invented names is large. A defender cannot keep a short blocklist of "fake names." An attacker has many candidates to register.
That points to a simple defense: check the name, not the list. Before installing, confirm the package exists, has a history, and is the one you meant.
What a team can do
- Verify before install. Follow how to check an AI suggested package before installing.
- Do not let agents install unchecked. See should AI coding agents run install commands.
- Use lockfiles and an internal registry or proxy. New names then have to pass a human step.
- Learn the attack. How slopsquatting exploits AI hallucinated package names walks through it.
Safeguard's SBOM and dependency scanning flags risky or unknown components in a project, and you can try it with the free scan at gold.safeguard.sh/scan.
What we could not confirm
This post relies on the abstract. It does not report the paper's breakdown by language or per model, and it does not name the models. Check the full paper for those.
Quick check
Does a 5.2% rate mean one in twenty installs is malicious?
No. It is the average share of invented package names in the paper's samples for commercial models. A name only becomes dangerous if someone registers it with bad intent.
Is the open-source figure of 21.7% a ranking of quality?
No. It is a group average under one study design. Do not use it to pick a tool.
Sources
Last reviewed: 19 October 2026.
