AI code assistants -- GitHub Copilot, Amazon CodeWhisperer, Cursor, Codeium, and a growing roster of alternatives -- have become standard development tools. GitHub reported in late 2024 that Copilot was generating over 40% of code in files where it was active. Developers overwhelmingly report productivity gains.
But security teams are raising legitimate concerns. AI-generated code carries risks that are qualitatively different from human-written code, and the security implications are just beginning to be understood.
Dependency Hallucination
One of the most concrete supply chain risks from AI code assistants is dependency hallucination: the tendency of language models to suggest package names that do not exist.
When an AI assistant suggests import useful-parser from 'string-toolkit-pro', a developer might install that package without verifying it exists or is the right choice. If the package does not exist, an attacker who notices the pattern can register it, creating a supply chain attack vector.
This is not theoretical. Researchers from Vulcan Cyber and academic institutions have documented the phenomenon:
- A 2024 study found that approximately 5% of package recommendations from popular AI code assistants referred to packages that did not exist in the respective package registries.
- Multiple instances of package name squatting based on AI hallucinations were observed on npm and PyPI.
- The names hallucinated by AI models are often plausible-sounding, making them attractive targets for typosquatting attacks.
The risk scales with AI adoption. As more developers use AI assistants and more code is AI-generated, the number of hallucinated package names increases, creating a larger attack surface for dependency confusion attacks.
Insecure Code Patterns
Multiple studies have examined the security quality of AI-generated code:
Stanford University (2023) found that participants using AI code assistants produced significantly less secure code than those coding without assistance, while believing their code was more secure. The AI assistant suggested insecure patterns (SQL injection, hardcoded credentials, missing input validation) that developers accepted without scrutiny.
NYU and Microsoft Research found that approximately 40% of AI-generated code samples contained security vulnerabilities, including buffer overflows, path traversals, and improper error handling.
OWASP's 2024 analysis of AI-generated web application code found persistent issues with:
- Insufficient input validation
- Missing authentication checks on sensitive endpoints
- Use of deprecated or insecure cryptographic functions
- Hardcoded secrets and default credentials in example code
- Missing CSRF protections
The fundamental issue is that language models learn from the corpus of existing code, which includes vast amounts of insecure code from tutorials, Stack Overflow answers, and legacy applications. The model does not distinguish between "this is how it is done" and "this is how it should be done."
The Training Data Problem
AI code assistants are trained on public code repositories, which means they inherit the security practices (good and bad) of the open-source ecosystem:
Vulnerable patterns are common in training data. The majority of code on GitHub does not follow security best practices. AI models learn from this majority and reproduce its patterns.
Outdated practices persist. Training data includes code from many years ago, when security practices were less mature. AI assistants may suggest patterns that were acceptable in 2015 but are now recognized as insecure.
Context is lost. A code snippet that is safe in its original context (a prototype, a tutorial, a test) may be unsafe when inserted into a production application. AI assistants do not understand the security requirements of the destination codebase.
Practical Risks for Organizations
For security teams, AI code assistants create several operational challenges:
Increased code review burden
AI-generated code requires the same (arguably more) security scrutiny as human-written code, but the volume of code being produced is increasing. Security review processes designed for human coding speeds may not scale to AI-assisted development rates.
SBOM accuracy
AI-generated code may pull in dependencies that were not part of the project's intended dependency tree. If SBOM generation happens at build time, these dependencies will be captured, but if the organization relies on manual dependency tracking, AI-introduced dependencies may be missed.
Compliance implications
Regulations like the EU CRA and FDA cybersecurity requirements mandate secure software development practices. If AI-generated code introduces vulnerabilities that a reasonable development process should have caught, the organization may face compliance questions about its development lifecycle.
License risks
AI assistants may generate code that is substantially similar to open-source code under restrictive licenses. If this code enters proprietary products without proper attribution, the organization faces legal risk. This is a supply chain transparency issue that SBOMs can help track.
Recommendations
Treat AI-generated code as untrusted input. Apply the same scrutiny you would to code from an unknown contributor. This means automated security scanning (SAST, SCA), code review, and testing.
Verify all dependency suggestions. Before installing a package suggested by an AI assistant, verify that it exists in the expected registry, is actively maintained, and has a reasonable security posture. Use lockfiles and package integrity verification.
Configure AI assistants with security context. Many AI assistants can be configured with organization-specific rules and patterns. Use this capability to steer the AI away from known-insecure patterns and toward your organization's security standards.
Include AI-generated code in your SBOM process. Ensure that your SBOM generation captures all dependencies, including those introduced by AI assistants. Monitor these dependencies for vulnerabilities like any other component.
Train developers on AI security risks. Developers need to understand that AI suggestions are probabilistic, not verified. Build a culture where AI suggestions are starting points, not finished products.
How Safeguard.sh Helps
Safeguard.sh provides the safety net that organizations need when AI code assistants are part of the development workflow. By generating comprehensive SBOMs and continuously monitoring dependencies for vulnerabilities, Safeguard catches the security issues that AI assistants introduce.
Safeguard's policy gates can flag suspicious dependencies -- packages with low download counts, recent publication dates, or no verified maintainers -- that may indicate hallucinated or malicious packages. Combined with vulnerability scanning and license analysis, Safeguard provides comprehensive visibility into the risks that AI-assisted development introduces into your software supply chain.