In 2021, researchers from NYU's Center for Cybersecurity ran GitHub Copilot against 89 real-world coding scenarios, generating 1,692 programs, and found that roughly 40% contained an exploitable weakness mapped to a known CWE category. That result predates the current generation of AI coding assistants embedded in nearly every major IDE, and the picture has not simply improved with scale — a later Stanford-led study found developers who used AI assistants wrote measurably less secure code, particularly around cryptography and SQL injection, while rating their own code as more secure than a control group's. A separate analysis of real Copilot-authored snippets pulled from public GitHub repositories, led by researcher Yujia Fu and collaborators, found 36% were vulnerable across 42 distinct CWE categories. Those are the well-studied risks. Less discussed are three others that engineering teams are only now confronting: assistants suggesting Python or JavaScript packages that don't exist and never did, developers pasting live API keys into a hosted chat window to debug a config file, and models trained on GPL and AGPL repositories reproducing copyleft code verbatim with no attribution. This post walks through each risk with the numbers behind it and what teams can actually do about it.
How often does AI-generated code actually contain vulnerabilities?
Frequently enough that "review normally" is not a sufficient control. The 2021 NYU paper, later presented at the IEEE Symposium on Security and Privacy, tested Copilot across scenarios drawn from the MITRE CWE "Top 25," and found that of the 1,692 programs generated, about 40% contained a bug that mapped to a CWE the researchers were specifically testing for — spanning issues like SQL injection, path traversal, and improper input validation. The more concerning follow-up finding came from a Stanford-led human study: developers given access to an AI assistant produced code with more security vulnerabilities in tasks involving string sanitization and cryptographic operations than a control group coding unassisted — and were simultaneously more confident their code was secure. That false-confidence gap is the real operational risk: a reviewer trusts code more, not less, because "the AI wrote it," at exactly the moment they should be scrutinizing it harder.
Can AI assistants suggest packages that don't exist?
Yes, and attackers are already exploiting it. A 2025 academic study generated 2.23 million code samples across 16 different LLMs and found that 19.7% referenced at least one package name that doesn't exist on the relevant registry — with open-source models hallucinating far more (21.7%) than commercial ones (5.2%). The attack this enables, dubbed "slopsquatting" by security researchers, is straightforward: if a model reliably suggests fast-json-parser or a similarly plausible-sounding package that isn't real, an attacker registers that exact name on PyPI or npm ahead of time, loaded with malware. Reporting from Socket.dev and Dark Reading on the same research noted that a meaningful share of hallucinated names studied had already been claimed by someone by the time researchers checked — some maliciously. The fix isn't a smarter prompt; it's verifying every new dependency against the real registry before pip install or npm install ever runs in CI.
How do AI coding assistants leak secrets?
The most common path is the simplest one: a developer pastes a config file, a .env, or a stack trace containing a live credential into a hosted chat-based assistant to get help debugging it, and that text leaves the organization's network the moment it's submitted. This isn't a hypothetical — it's the exact scenario that led Samsung to ban internal ChatGPT use in 2023 after engineers pasted proprietary source code into the tool while troubleshooting. IDE-embedded assistants reduce but don't eliminate this: autocomplete suggestions are typically generated from local context, but chat panels and "explain this error" features frequently send the surrounding file, including anything hardcoded in it, to a remote API. The credential doesn't need to be exfiltrated by the model to become a real incident — it's already left your perimeter once it's in a third-party provider's request logs. The established mitigation is not to rely on the assistant to redact secrets; it's to make sure no live credential exists in the file being shared in the first place.
Why is license contamination a real legal risk, not just a technical one?
Because models like Copilot and Codex are trained on large volumes of public GitHub code, some of it under GPL and AGPL licenses that require attribution or, in stronger cases, that derivative works be released under the same terms. When a model reproduces a memorized snippet verbatim — which independent researchers have documented happening, particularly for widely-duplicated code — a team merging that suggestion into a proprietary or commercially-licensed codebase can unknowingly take on a copyleft obligation. This is not a theoretical concern raised only by security teams: it's the substance of the Doe v. GitHub litigation filed in 2022 over Copilot's training and output, which as of this writing continues to work through the courts and has kept the question of AI-output license liability squarely in view for legal and engineering teams alike. Unlike a security bug, a license contamination issue often isn't found by testing — it surfaces during an acquisition's due diligence or a customer's vendor audit, months or years after the code shipped.
What should engineering teams actually do about this?
Treat AI-generated code the same way you'd treat a contribution from an unvetted external contractor: verify before merge, not after. For secret leakage, the most reliable control is scanning at the point code leaves the developer's machine, not trusting the assistant. Safeguard's pre-push git hook scans new commits locally before they leave the laptop, checking against 200+ issuer-specific patterns (AWS, GitHub, Stripe, OpenAI, Anthropic, and more) and verifying live credentials against the issuing service so a finding isn't just a guess — it's confirmed exploitable, with a revoke-and-rotate playbook attached. For license contamination, Safeguard's license compliance engine classifies every dependency's obligations (permissive, weak copyleft, strong copyleft, network copyleft) and enforces an allow/deny policy at PR time, so a GPL-licensed snippet or dependency introduced through an AI suggestion gets flagged before it merges, not during due diligence. Neither insecure-suggestion detection nor package-hallucination checking is a purpose-built Safeguard capability today, which is exactly why they need to sit inside the same reviewed-and-gated PR process as everything else: mandatory code review, dependency verification against the real registry, and SAST scanning that doesn't care whether a human or a model wrote the line it's flagging.