In April 2025, security researchers at the University of Texas at San Antonio published a study analyzing 576,000 code samples generated by 16 large language models and found that AI models hallucinate package names in roughly 5.2% of outputs from commercial models and 21.7% from open-source models — inventing dependencies that don't exist, but that attackers can register and weaponize the moment a name leaks into a popular training set. This is the new shape of software supply chain risk: not just vulnerable open-source components, but code whose provenance, dependencies, and even existence were fabricated by a model. GitHub reported in 2024 that AI now writes on the order of 30-46% of code in enabled repositories. Traditional composition analysis tools, including Black Duck, were built to catalog known open-source packages against a knowledgebase — a model that assumes the code and its dependencies already exist somewhere to be matched. AI-generated code breaks that assumption. Here's what security teams need to understand, and where legacy SCA tooling runs out of road.
Why is AI-generated code a supply chain security problem, not just a code-quality problem?
Because AI models don't just write insecure logic — they invent entire dependency chains that never existed, turning a naming hallucination into an installable attack. When a developer asks Copilot, Cursor, or ChatGPT to solve a problem, the model frequently suggests importing a package to do it. The UTSA/University of Oklahoma/Virginia Tech study (Spracklen et al., "We Have a Package for You!", USENIX Security 2025) ran 30 different prompts across 16 models and found 205,000 unique hallucinated package names. Worse, 43% of hallucinated names repeated across multiple runs, meaning attackers don't need to guess — they can query the same models researchers used, register the hallucinated name on PyPI or npm before anyone else does, and wait for developers (or their AI assistants) to pip install malware. This attack pattern now has a name: slopsquatting, coined in 2025 as a direct evolution of typosquatting for the AI era. A traditional SCA scan of the resulting lockfile shows a "known" package with a plausible name and a real (malicious) version — there is no CVE to flag and no license anomaly to catch, because the package is brand new and was never vulnerable, it was purpose-built to look normal.
How much AI-generated code is actually shipping into production codebases right now?
A lot, and the share is climbing every quarter. GitHub's own 2024 data put Copilot's suggestion-acceptance contribution at roughly 30% of code in files where it's enabled, and internal enterprise telemetry shared at RSA 2025 put some engineering orgs above 40%. Stack Overflow's 2024 Developer Survey found 76% of developers were using or planning to use AI coding tools, up from 70% the year before, while trust in the accuracy of that output actually fell — only 43% said they trusted AI-generated code, down from 40% the prior year but with confidence in complex tasks dropping much further. GitClear's 2024 analysis of 211 million changed lines of code found that AI-era commits show an eight-fold increase in duplicated code blocks compared to 2020, and a marked decline in refactoring — code is being pasted in and shipped, not restructured or reviewed with the same rigor as hand-written PRs. That combination — more AI code, less scrutiny, more duplication — is exactly the environment where an SCA tool tuned to spot known-bad components misses the real risk sitting in the diff itself.
What specific vulnerability classes show up disproportionately in AI-generated code?
Three: hallucinated dependencies, insecure-by-default patterns, and license/provenance gaps — and all three sit outside what binary and package matching was designed to catch. On the insecure-pattern front, a 2024 Stanford study extending earlier Copilot research found that developers using AI assistants were more likely to introduce SQL injection, hardcoded credentials, and weak cryptographic defaults than those coding unassisted, and — notably — were also more confident their code was secure, a false sense of security the researchers called out explicitly. Snyk's 2023 AI code security report found similarly that 56% of developers using AI coding tools said security wasn't a top priority when using them, even though 79.9% admitted their AI tool had suggested insecure code before. On provenance, models trained on public repositories reproduce license-encumbered snippets verbatim often enough that GitHub added a code-reference filter to Copilot specifically to flag matches against public code — an admission that attribution and license risk is real and ongoing, not hypothetical.
Why do SCA tools built around Black Duck's knowledgebase model struggle here?
Because they're architected to answer "is this known component vulnerable?" — and AI-generated risk is dominated by components and code that aren't known at all. Black Duck's core value proposition, inherited from decades as Synopsys' composition analysis product, is a knowledgebase of tens of millions of open-source projects matched via binary and snippet signatures, cross-referenced against CVE and license databases. That's a strong answer to "did we pull in a vulnerable version of Log4j," and it remains necessary. But a slopsquatted package with no CVE history, a first-seen dependency published last week, or a function that's syntactically novel but semantically identical to a known-bad pattern doesn't trip a knowledgebase match — there's nothing in the database to match against yet. The result is a coverage gap that widens every month AI-assisted development grows relative to human-authored, reviewed code, because SCA was built for a world where the median dependency is old and well-known, and AI-era development produces a long tail of dependencies and code paths that are neither.
What controls actually reduce risk from AI-generated code today?
Four controls consistently show up in guidance from OWASP, NIST, and CISA in 2024-2025: dependency provenance verification before install, not after; policy-as-code gates in CI that block unpinned or newly-published packages by default; mandatory human review weighted toward AI-authored diffs specifically; and continuous re-scanning of production dependencies rather than point-in-time audits. NIST's AI Risk Management Framework generative AI profile (NIST AI 600-1, published July 2024) explicitly calls out "confabulation" — its term for hallucination — as a risk category requiring verification controls before AI outputs are trusted in production workflows. OWASP's 2025 Top 10 for LLM Applications added supply chain risks tied to model-generated artifacts as a named category (LLM03) for the first time. Practically, that means CI pipelines need a gate that asks "was this package published in the last N days, with insufficient download history and no reputation signal" before install — a check that has nothing to do with whether the package matches a known-vulnerable signature, and everything to do with whether it should be trusted at all.
How Safeguard Helps
Safeguard was built for exactly this gap: securing code and dependencies whose risk profile can't be resolved by matching against a static knowledgebase. Where legacy SCA answers "is this a known-vulnerable open-source component," Safeguard's platform is designed to evaluate provenance and trust signals for every dependency at the moment it's introduced — flagging newly-published packages, low-reputation registries, and naming patterns consistent with slopsquatting before they ever reach a lockfile, not after a CVE is filed months later.
For AI-generated code specifically, Safeguard applies policy-as-code gates in CI/CD that treat AI-authored diffs as a distinct risk class: requiring dependency freshness and reputation checks, flagging hardcoded secrets and weak crypto defaults that studies show AI assistants introduce at elevated rates, and routing AI-heavy changes toward mandatory review rather than auto-merge. Combined with continuous monitoring of production dependencies — so a package that looked clean at merge time but is later reported for typosquatting or compromise gets caught on an ongoing basis, not at the next annual audit — Safeguard closes the specific gap that knowledgebase-driven SCA tools like Black Duck leave open: securing the code and packages that don't have a known history yet, because a model just invented them.
If your engineering org has crossed the point where a third or more of new code is AI-assisted — and GitHub's numbers suggest most have — the question isn't whether your existing SCA tool is bad, it's whether it was ever built to answer the question you now need answered. Talk to Safeguard about extending your supply chain security program to cover provenance, freshness, and AI-specific risk patterns alongside the CVE and license coverage you already have.