Is ChatGPT safe for writing code? Benefits and security risks

AI chat assistants such as ChatGPT can write useful code quickly, and they can also produce vulnerable code, invent packages and receive data you should not paste. Treat the output as an untrusted contribution from a fast junior colleague: review it, test it, scan it and keep secrets out of prompts.

What are the benefits?

The practical gains are speed on boilerplate, explanations of unfamiliar code, test scaffolding and first drafts of small functions. For a developer facing a library they have never used, a working example in seconds beats an hour of reading. We are not citing productivity figures, because published numbers vary by task and method.

The risk comes from the same property. A fluent answer looks finished, so it gets less scrutiny than code a teammate wrote.

What does research say about the security of generated code?

The best known study is "Asleep at the Keyboard?" by Pearce and colleagues, published at the IEEE Symposium on Security and Privacy in 2022. The authors prompted GitHub Copilot, a code completion model, with 89 scenarios built around high-risk weakness types from MITRE's CWE Top 25. That produced 1,689 programs, of which approximately 40 percent were found vulnerable.

Read it carefully. It studied one tool at one point in time, not ChatGPT, and models have changed since. What carries over is the method and the lesson: a model trained on public code has seen insecure code, so it can reproduce it, especially when the prompt does not mention security.

Which risks apply to AI assistants as applications?

OWASP's Top 10 for LLM and generative AI applications (2025) names the risks. Several matter for developers using chat tools: prompt injection, where inputs alter model behaviour; sensitive information disclosure, which can affect both the model and its application; supply chain vulnerabilities; improper output handling, meaning insufficient validation of what a model returns before it is used; excessive agency for systems granted the power to act; and misinformation, where a model states something false with confidence.

For coding, improper output handling and misinformation are the daily ones. Code from a model is output that needs validation before it runs, and a plausible but wrong claim about an API or library is common enough that you must check.

What goes wrong in practice?

Insecure patterns. String built SQL queries, missing authorization checks, weak cryptographic choices, disabled certificate verification copied from a forum example.

Hallucinated or outdated dependencies. The assistant names a package that does not exist or recommends an old version with known flaws. An attacker who registers a made up name can turn a hallucination into a supply chain compromise, so verify each package exists and is the one you intend.

Data leakage. Developers paste source code, credentials or customer data into a prompt. Check your organisation's rules and the provider's data terms before pasting anything sensitive.

Overtrust. Reviewers skim AI generated changes because they look tidy.

A constructed example

A developer asks an assistant for an endpoint that returns an invoice by ID. The answer works and has clean error handling. It fetches the invoice by the ID in the URL and returns it. It never checks that the invoice belongs to the logged in customer.

A code reviewer who reads the diff as ordinary code might catch this. One who assumes the assistant handled security might not. An automated check in the pipeline, plus a reviewer checklist item reading who is allowed to see this object, makes the catch routine rather than lucky.

How should teams use these tools safely?

NIST's SP 800-218A extends the Secure Software Development Framework with practices specific to AI model development, aimed at producers of AI models and systems and at those who acquire them. Its core message for adopters is the same as for any software: apply secure development practices to what you build and buy, including AI.

Put the assistant inside your normal controls. Run static analysis and dependency checks on every change, whoever wrote it. Safeguard's SAST, dependency scanning with reachability and policy gates on pull requests apply to AI written code the same way as human code, and its MCP server lets assistants query that security context while they work. MCP Guardrail can limit what an agent may call.

How to check or apply this

1. Write a short rule on what may and may not be pasted into prompts.
2. Require human review of AI generated code to the same standard as any change.
3. Add an authorization question to the review checklist.
4. Verify that every suggested package exists, is maintained and is the intended one.
5. Run SAST, secrets and dependency scans in the pipeline on every pull request.
6. Prompt for security explicitly, then still verify the result.
7. Log which tools are approved and review the list each quarter.

Quick check

Sources

• Pearce et al., Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions
• OWASP Top 10 for LLM and Generative AI Applications
• NIST SP 800-218A, Secure Software Development Practices for Generative AI and Dual-Use Foundation Models
• NIST SP 800-218, SSDF

Last reviewed: 18 October 2026.