In 2021, researchers at NYU ran 89 security-relevant coding scenarios through GitHub Copilot and found that roughly 40% of the resulting completions contained a vulnerability mapping to a CWE Top-25 category — SQL injection, hardcoded credentials, and buffer-overflow-prone patterns among them. Three years later, a 2024 academic analysis of 576,000 AI-generated Python and JavaScript code samples across 16 different LLMs found the models collectively invented 205,474 unique package names that don't exist on PyPI or npm — an average hallucination rate of 5.2% for commercial models and 21.7% for open-source ones. Neither finding is a fluke or a solved problem; both describe structural properties of how these models were trained and how they generate output. AI coding assistants are now writing a meaningful share of production code at most software companies, and they carry forward the same vulnerability patterns, deprecated APIs, and leaked credentials that appear in their training data — at the speed and volume of autocomplete. This piece catalogs the recurring pitfalls — hardcoded secrets, insecure defaults, outdated API usage, hallucinated packages, and misplaced developer trust — and the concrete mitigations that catch each one before it reaches production.
Do developers actually write less secure code with AI assistance?
Yes, and a widely cited Stanford study put a number on it. Researchers Neil Perry, Megha Srivastava, Deepak Kumar, and Dan Boneh ran a controlled study (published at CCS) comparing developers who used an AI code assistant against a control group writing the same tasks unassisted. The AI-assisted group produced significantly more insecure solutions — including SQL injection and flawed string-encryption implementations — than the control group. The more striking result was psychological, not technical: participants who used the AI assistant were more likely to believe their code was secure than those who didn't, even though their code measurably wasn't. The authors describe this as a form of automation complacency — the assistant's confident, well-formatted output reads as authoritative, so developers skip the scrutiny they'd apply to their own first draft. That gap between perceived and actual security is arguably the more dangerous pitfall, because it suppresses the code review behavior that would normally catch the other four pitfalls in this list.
How often does generated code contain exploitable vulnerabilities?
Frequently enough that "occasionally" undersells it. The NYU study by Pearce et al. ("Asleep at the Keyboard?", 2021) is still the most-cited data point: across 89 scenarios spanning MITRE's CWE Top 25, roughly 40% of Copilot's completions contained a vulnerability a static analyzer could confirm as potentially exploitable. The scenarios weren't obscure edge cases — they covered common patterns like building SQL queries from user input, deserializing untrusted data, and generating cryptographic keys. The underlying cause is training data, not model malice: these models learned from billions of lines of public code, and public code includes an enormous volume of tutorials, Stack Overflow answers, and abandoned side projects that were never written with production security in mind. A model optimizing for "code that looks like what a human would plausibly write next" will faithfully reproduce insecure idioms as often as they appear in its training corpus.
What is package hallucination, and why is it a supply-chain risk?
Package hallucination is when an LLM confidently recommends installing a dependency that doesn't exist, and it has become a named attack surface — "slopsquatting" — because attackers can register the invented name before a developer does. The 2024 study referenced above analyzed 576,000 generated code samples and found average hallucination rates of 5.2% for commercial models and 21.7% for open-source models, totaling over 205,000 unique fake package names. This isn't hypothetical: a security researcher demonstrated the exploit path by publishing an empty package under a name that an LLM had hallucinated, huggingface-cli — a name resembling Hugging Face's real huggingface_hub CLI tooling — and it accumulated over 30,000 downloads in three months, proof that developers copy pip install or npm install commands straight out of AI suggestions without checking the registry first.
Which specific coding pitfalls show up most often in generated snippets?
Beyond the two research-backed patterns above, four pitfalls recur constantly in day-to-day AI-generated code. Hardcoded secrets: models trained on public repositories containing leaked API keys and credentials reproduce that pattern when asked to "write a script that connects to AWS" — the fastest correct-looking answer includes a literal key rather than a reference to an environment variable or secrets manager. Insecure defaults: generated boilerplate frequently disables TLS certificate verification, opens permissive CORS ( Access-Control-Allow-Origin: * ), or stands up an endpoint with no authentication, because that's the shortest path to a runnable demo. Outdated API usage: training data has a cutoff, so models keep suggesting deprecated or weakened primitives — MD5 for password hashing, DES for encryption, or library calls that were superseded by a safer replacement years ago. And missing input validation: generated request handlers often skip sanitization entirely, assuming a "happy path" that a fuzzing pass or a real attacker will not respect.
How can teams catch hardcoded secrets before an AI-assisted commit ships?
The fix has to happen at the same speed the problem is created — inline, before the commit lands — because reviewing every AI suggestion by eye doesn't scale. Safeguard's secrets scanning runs issuer-specific pattern matching across source, git history, container layers, and — notably for AI workflows — model weights, since fine-tuned models can bake tokens and keys directly into their parameters. Rather than just pattern-matching, Safeguard verifies each candidate secret against the issuing service (an AWS key against sts:GetCallerIdentity, a GitHub token against /user) so a flagged finding means the credential is live, not a maybe. A pre-push git hook ( safeguard install-hook --hook pre-push ) catches AI-suggested credentials at the exact moment a developer commits a "just get it working" snippet, before it ever reaches a shared branch or a build log.
Can policy enforcement catch hallucinated packages before install?
Yes, if the check happens at the point where a package name turns into an install command rather than after the fact. Safeguard ships a built-in "Typosquat detected" guardrail that blocks any package whose name falls within edit-distance 2 of a top-1000 package but isn't published by the expected maintainer — the same structural pattern a slopsquatted, LLM-hallucinated package name exhibits when it closely resembles a real, popular library. Because guardrails evaluate at CI, registry, and admission time as policy-as-code rather than a one-off manual review, a hallucinated dependency an AI assistant suggested in an editor gets a second, automated check before it's pinned into a lockfile — closing the gap between what a model confidently recommends and what a team should actually trust.