By March 2023, GitHub Copilot had over 1 million paying users, and ChatGPT was being used by millions of developers for code generation. Amazon CodeWhisperer was generally available. Google was preparing to launch Bard with code generation capabilities. The era of AI-assisted coding wasn't coming — it had arrived.
The security community was starting to grapple with an uncomfortable question: what happens when a significant percentage of production code is written by machines trained on a corpus that includes vulnerable code?
The Research Says It's a Problem
A landmark study from Stanford University (published December 2022, widely discussed in early 2023) found that participants who used AI code assistants wrote significantly less secure code than those who didn't. More concerning, participants using AI assistants were more likely to believe their code was secure.
The study tested participants on common security tasks like implementing cryptographic operations, SQL queries, and memory management. Across the board, AI-assisted participants produced more vulnerabilities:
- More SQL injection vulnerabilities
- More insecure cryptographic implementations
- More buffer overflow conditions
- More insecure use of APIs
The key insight wasn't that AI generates bad code — it was that AI generates code that looks correct and professional, making developers less likely to scrutinize it for security issues.
Five Concrete Security Risks
1. Vulnerable Code Patterns From Training Data
AI models learn patterns from existing code. The training data includes millions of code samples from public repositories, Stack Overflow answers, and documentation. A lot of that code contains vulnerabilities.
When a developer asks Copilot for help with XML parsing, the model might generate code using a library configured without protection against XML External Entity (XXE) attacks. The code works. The tests pass. The vulnerability goes undetected until someone exploits it.
Real examples observed in early 2023:
- Insecure deserialization: AI-generated code using
pickle.loads()on untrusted input - Hardcoded secrets: Example code with placeholder API keys that developers don't always replace
- Weak cryptography: Code using MD5 or SHA-1 for security-sensitive operations
- Missing input validation: Generated code that processes user input without sanitization
2. Dependency Hallucinations
When AI generates code that requires packages, it sometimes references packages that don't exist. This creates a supply chain attack vector known as "package hallucination" or "AI package confusion."
Security researchers demonstrated this by:
- Querying ChatGPT for code that required specific packages
- Noting packages that were consistently recommended but didn't exist
- Registering those package names on npm/PyPI
- Observing installation attempts from developers who followed ChatGPT's suggestions
The attack works because the AI recommendations feel authoritative. A developer who might double-check a random package recommendation from a forum post is less likely to question a recommendation from an AI tool they trust.
3. Outdated and Deprecated APIs
AI models have knowledge cutoffs. Code generated in 2023 might use APIs that were deprecated or found to be insecure after the training data cutoff. The model doesn't know about:
- New CVEs discovered after training
- API deprecations
- Security patches that changed function signatures
- Libraries that have been abandoned
This creates a time-lag vulnerability where AI-generated code is systematically behind the current security state of the libraries it uses.
4. Context-Inappropriate Security Levels
AI code generation lacks understanding of your specific security requirements. It generates "average" code — appropriate for a tutorial but not for a banking application. Examples:
- Using basic authentication when your application requires OAuth 2.0
- Generating logging code that includes sensitive data
- Creating database connections without SSL/TLS
- Implementing file handling without proper permission checks
The AI doesn't know that the code it's generating will handle medical records, financial transactions, or classified data.
5. License Compliance Risks
AI models trained on open-source code may generate code that's substantially similar to code under restrictive licenses (GPL, AGPL). This creates legal risks that are also supply chain risks — using improperly licensed code can force disclosure of proprietary code or result in litigation.
Several lawsuits filed in early 2023 against GitHub Copilot and OpenAI raised exactly these concerns.
The Organizational Challenge
The security risks of AI-generated code are compounded by organizational dynamics:
Speed pressure: Developers adopt AI tools because they increase productivity. Security review slows things down. The productivity gains from AI code generation create pressure to reduce security review overhead, exactly when it needs to increase.
Skill erosion: As developers rely more on AI for code generation, their ability to identify security issues in generated code may atrophy. Junior developers who learn to code with AI assistance may never develop the security instincts that come from writing code from scratch.
Audit trail gaps: It's often unclear which code was AI-generated and which was human-written. Without this distinction, security teams can't apply appropriate scrutiny to AI-generated code.
Practical Mitigations
Treat AI Output as Untrusted Input
Every line of AI-generated code should pass through:
- Static Application Security Testing (SAST)
- Software Composition Analysis (SCA) for any dependencies
- Human security review for security-sensitive components
Verify Every Dependency
Before installing any package recommended by an AI tool:
- Confirm the package exists on the intended registry
- Check the package's maintenance status, download count, and publisher
- Scan the package for known vulnerabilities
- Verify it's not a typosquat
Implement Guardrails
Configure your development environment to:
- Run linters and security scanners automatically
- Block commits that introduce known vulnerable dependencies
- Flag AI-generated code patterns known to be insecure
Track AI-Generated Code
Establish processes to identify and track which code was AI-generated. This allows security teams to prioritize review of AI-generated code and measure the effectiveness of AI security mitigations.
How Safeguard Helps
Safeguard is designed to catch exactly the kinds of issues that AI code generation introduces:
- Dependency Validation: Safeguard verifies that every dependency in your project is legitimate, actively maintained, and free of known vulnerabilities — catching hallucinated packages and outdated recommendations.
- Continuous Vulnerability Scanning: As new CVEs are discovered in libraries that AI tools have been recommending, Safeguard alerts you immediately across all projects.
- SBOM-Based Tracking: Safeguard generates comprehensive SBOMs that give you full visibility into what's in your software, regardless of whether it was added by human or AI.
- License Compliance Analysis: Safeguard identifies license risks in your dependency tree, catching potential compliance issues from AI-recommended packages.
AI code generation is here to stay, and it will only become more pervasive. The organizations that build security guardrails around AI-generated code today will avoid the breach headlines of tomorrow.