When a developer asks an AI coding assistant to "write a function that logs in a user," the model doesn't reason about authentication best practices from first principles. It predicts the most statistically likely sequence of tokens based on billions of lines of code it was trained on — code scraped from public repositories, forums, and tutorials that were never audited for security. Much of that code is old, unmaintained, or was written by developers solving a narrow problem without threat modeling. Some of it contains the exact vulnerability patterns security teams have spent two decades trying to eradicate: string-concatenated SQL queries, hardcoded secrets, disabled TLS verification, and weak cryptographic defaults.
The result is a feedback loop: insecure code on GitHub trains models that generate insecure code, which developers commit back to GitHub, which trains the next generation of models. Understanding how this propagation happens — and where it can be intercepted — is now a core software supply chain security problem, not just an academic curiosity.
Why do AI coding assistants keep suggesting the same vulnerabilities?
Because they were trained to reproduce patterns, not to evaluate risk. In the 2021 NYU study "Asleep at the Keyboard," researchers ran GitHub Copilot against 89 coding scenarios mapped to MITRE's Top 25 CWEs and found that roughly 40% of the generated programs contained exploitable vulnerabilities — including SQL injection (CWE-89), path traversal (CWE-22), and out-of-bounds writes (CWE-787). A 2023 Stanford follow-up, "Do Users Write More Insecure Code with AI Assistants?" (Perry et al., IEEE S&P), went further: developers who used an AI assistant wrote measurably less secure code than a control group across five programming tasks, and — more concerning for defenders — they were more likely to believe their AI-assisted code was secure. The model doesn't "know" that eval()-ing user input is dangerous; it has simply seen that pattern thousands of times in Stack Overflow snippets from 2012 that never anticipated adversarial input.
How much of a model's training data actually contains insecure code?
Enough that it shows up reliably in output, even though nobody knows the exact percentage. Public code corpora used to train large language models — GitHub public repos, The Stack, competitive programming archives — were never security-labeled at scale, and static analysis of GitHub's public corpus consistently finds that a meaningful share of repositories carry at least one known CWE-class flaw, from unpatched dependency versions to plaintext credential handling. Training pipelines generally deduplicate and filter for license and quality, but not for security posture, because there is no reliable automated ground truth for "this snippet is safe." A 2023 study on LLM-based code generation found that models trained predominantly on older code corpora reproduce deprecated and vulnerable API calls (like MD5 for password hashing or pickle.loads() on untrusted input) at rates matching their prevalence in the training set — the model has no mechanism to know that guidance changed after its training cutoff.
Can someone deliberately poison a model's training data to plant vulnerabilities?
Yes, and it doesn't take a large volume of poisoned samples to work. OWASP's Top 10 for LLM Applications (2025 revision) lists "Training Data Poisoning" as LLM04, and academic red-team research on code-generation models has shown that inserting a small number of carefully crafted malicious examples — sometimes fewer than 100 out of millions of training samples — into a fine-tuning or pretraining set can reliably bias a model toward suggesting a specific insecure pattern (like a backdoored authentication check or a weakened cipher default) when a particular trigger phrase appears in the prompt. This matters directly for software supply chain security because public package repositories, tutorial sites, and even Stack Overflow answers are all plausible ingestion sources for model training, and none of them enforce provenance or integrity guarantees on the code they host. An attacker doesn't need to compromise a model vendor's infrastructure — publishing enough convincingly-styled poisoned examples in the open is a viable, low-cost attack surface.
Why do LLMs recommend software packages that don't exist?
Because they generate plausible-sounding names from statistical patterns rather than checking a real package index, and attackers have already weaponized this. Research from Socket and academic collaborators (the 2024 paper "We Have a Package for You!") tested 16 code-generating LLMs across roughly 576,000 code samples and found an average package hallucination rate of 5.2% for commercial models and 21.7% for open-source models — meaning open-source models suggested a non-existent package roughly one in five times when asked to import a dependency. Attackers monitor these predictable hallucinated names and pre-register them on PyPI and npm as malicious packages, a technique now called "slopsquatting." A developer who copies an AI suggestion without verifying the package exists and is legitimate can pull malware directly into their build — turning a model's training-data blind spot into an active supply chain compromise.
Are newer and larger models actually producing more secure code?
Marginally, on narrow benchmarks, but not enough to change the risk calculus. Vendors have started applying reinforcement learning and secure-coding fine-tuning on top of base models, and some 2024-2025 benchmark results (including those tracked by Veracode's State of Software Security research) show incremental improvement in avoiding the most obvious CWE Top 25 patterns. But the same research also finds that model size and recency don't reliably predict secure output for less common but still critical patterns — insecure deserialization, SSRF-prone URL handling, and race conditions in concurrent code remain common across model generations, because these patterns are underrepresented and inconsistently labeled in training corpora. Larger models are also more fluent at producing code that "looks" secure — proper variable names, comments explaining the logic — while still containing the underlying flaw, which makes manual review harder, not easier.
Does this mean AI-generated code is riskier than human-written code?
Not inherently riskier, but differently risky, and organizations that treat it identically to human code miss the gap. Human developers write insecure code too, but they can be trained, held accountable in code review, and their velocity is bounded. AI assistants generate at a scale and speed that can outpace existing review processes, and they suggest vulnerable patterns with consistent, repeatable confidence — a subtle prompt variation can produce the same flawed authentication check across dozens of files in a single session. The practical implication for engineering organizations adopting Copilot, Cursor, Claude Code, or similar tools isn't to ban AI-assisted development; it's to treat every AI-generated commit as untrusted input to the software supply chain until it's verified, the same way you'd treat a third-party dependency pulled from an unfamiliar registry.
How Safeguard Helps
Safeguard is built for exactly this shift: AI-generated code is now a first-class source of software supply chain risk, and it needs the same provenance, integrity, and continuous verification controls as any external dependency. Safeguard's platform sits in the SDLC to catch the patterns described above before they reach production:
- Pattern-aware SAST tuned for AI-generated code. Safeguard's static analysis engine is calibrated against the specific CWE classes that AI assistants reproduce most often — injection flaws, hardcoded secrets, weak cryptography, and insecure deserialization — flagging them at commit and PR time regardless of whether a human or a model wrote the line.
- Dependency and package verification. Because hallucinated package names are now an active attack vector, Safeguard validates every new dependency introduced in a PR against real registry data and known-malicious package intelligence, closing the exact gap that slopsquatting exploits.
- SBOM-backed provenance tracking. Safeguard maintains a continuously updated software bill of materials so that when a new CVE or a newly disclosed poisoned package surfaces, teams can immediately see whether AI-suggested code paths brought it into the codebase.
- DAST validation as a second gate. Static review alone can't catch every logic-level flaw introduced by fluent-but-wrong AI suggestions. Safeguard's dynamic analysis exercises running applications to catch exploitable behavior that looks clean in source but fails under adversarial input.
- SOC 2-aligned audit trails. Every finding, override, and remediation is logged, giving compliance and security teams a defensible record of how AI-assisted code was vetted — a requirement that's quickly becoming standard in vendor security questionnaires and SOC 2 audits.
The uncomfortable truth about LLM training data is that it encodes the internet's coding history, mistakes included, and no amount of prompt engineering fully compensates for that at the point of code generation. The fix isn't a smarter prompt — it's a verification layer that doesn't trust code based on who or what wrote it. That's the layer Safeguard provides.