In 2021, a team at NYU's Center for Cyber Security ran GitHub Copilot against 89 programming scenarios covering MITRE's top weakness categories and found that roughly 40% of the 1,692 resulting programs contained security weaknesses mappable to a CWE — everything from SQL injection to path traversal (Pearce et al., "Asleep at the Keyboard?"). That study is now five years old and the models have changed, but the pattern has not disappeared: replication work since then (arXiv 2310.02059, arXiv 2311.11177) still finds meaningful vulnerability rates that vary by language and weakness class rather than a clean trend toward zero. Whether your SAST tooling and DAST tooling can actually catch these defects is a separate question from whether a vendor claims it does — and it's the one this post is really about. Snyk's 2023 AI-Generated Code Security Report adds the organizational half of the story — more than half of practitioners said AI-generated code sometimes or frequently introduced security issues, yet over 75% still believed that code was more secure than what a human wrote, and roughly 80% admitted bypassing their own security policies under deadline pressure to ship it anyway. That gap between confidence and outcome is the actual risk. This post lays out a concrete framework for evaluating AI codegen tools and gating their output before it merges, rather than trusting a model's self-reported correctness.
Why can't you trust a code-generation model's own confidence signal?
Because a language model's fluency is not evidence of correctness — it is evidence that the output resembles code the model has seen, and confident-sounding code can still be wrong in exactly the ways that matter for security. The NYU study's most important finding wasn't the 40% rate itself; it was that Copilot's insecure completions were often syntactically clean and idiomatic, with no surface signal distinguishing a safe SQL parameterization from a vulnerable string-concatenated query. A recent line of research on iterative and agentic code generation (arXiv 2506.11022, "Security Degradation in Iterative AI Code Generation") goes further, documenting that multi-turn generation — the pattern used by coding agents that iterate on their own output — can progressively degrade security posture across turns rather than converge toward a safer solution. That means asking the model to "review its own code" or "fix the bug" is not a substitute for independent verification; it's another generation step with the same failure mode.
What does treating AI output as an untrusted contributor actually mean?
It means every line an AI tool produces goes through the same gates you'd apply to a pull request from a contributor you've never worked with before — not a lighter version of review because "the tests passed." Concretely: static analysis that traces data flow from source to sink (not just pattern-matches dangerous function names), dependency and software composition analysis on anything the model imports or adds to a lockfile, and secrets scanning on the diff, since models trained on public code will confidently reproduce hardcoded API keys and default credentials seen in training examples. None of these checks care who or what authored the code, which is exactly the point — they evaluate the artifact, not the author's claimed intent. Safeguard's SAST engine implements this by tracing untrusted input from a source (a request parameter, a CLI arg) to a dangerous sink (a SQL query, a command exec) across functions and files, producing a dataflow trace rather than a bare line number, per its application security testing documentation. The same logic extends past static analysis: for AI-generated code that ships as a running service, DAST tooling catches runtime misconfigurations and exposed endpoints that static analysis alone was never designed to see, which is why pre-merge SAST should be paired with post-deploy dynamic scanning rather than treated as a substitute for it.
How does AI-assisted code change the maintainability side of the risk equation?
It shifts a meaningful share of changed code from refactored, reviewed logic to duplicated blocks that no one owns, which compounds the security review burden downstream. GitClear's 2025 analysis of 211 million changed lines from 2020 to 2024 found that copy-pasted (cloned) code rose from 8.3% to 12.3% of all changed lines over that period, while refactoring-attributed lines fell from roughly 25% to under 10% — and duplicated code blocks specifically rose roughly eightfold in 2024, the year AI-assisted commits became widespread. Duplicated logic means a fix for a vulnerable pattern in one place doesn't propagate to its copies elsewhere; a reviewer who approves one instance has no visibility into the six pasted variants scattered across the codebase. This is a quality problem that becomes a security problem the moment one of those copies handles untrusted input differently than the original did.
What should an evaluation framework for a codegen tool actually measure?
It should measure defect rate on security-relevant code paths, not general helpfulness or how often suggestions are accepted. A workable evaluation checks four things before adopting or continuing to use a tool: (1) vulnerability injection rate on a fixed benchmark of security-sensitive tasks — auth, deserialization, file handling, SQL construction — re-run whenever the underlying model changes; (2) dependency hygiene, meaning whether the tool suggests packages with known CVEs, abandoned maintenance status, or names close enough to popular libraries to suggest typosquat risk; (3) secret and credential hygiene in generated fixtures, config, and test code; and (4) degradation under iteration — whether asking the tool to "fix" or "extend" its own prior output holds security properties steady or erodes them, per the arXiv 2506.11022 findings above. None of these require trusting the vendor's marketing; they're measurable with the same SAST tooling, SCA, and secrets scanning you'd run on any other code.
What does a concrete pre-merge gate for AI-generated code look like?
It looks like a policy-as-code rule evaluated in CI, blocking the merge automatically rather than relying on a reviewer to notice. A minimal gate combines a blocking condition on any newly introduced critical- or high-severity CWE finding, a check that no new dependency lacks an SBOM entry or carries a known-exploited CVE, and a secrets scan on the diff — all evaluated the same way regardless of whether a human or a model authored the change. Safeguard's guardrails model expresses this as YAML policy with BLOCK, WARN, or AUTO_FIX effects evaluated against SBOM, vulnerability, and reachability data, and its safeguard gate CLI step exits non-zero on any blocking match so the PR simply cannot merge until the finding is resolved or an approver grants a time-boxed, audited exception. The mechanism doesn't care that the diff came from an AI assistant — which is exactly why it works as a gate rather than a suggestion.
How should a team weigh AI codegen's productivity gain against this risk?
By keeping the productivity gain and pricing the risk separately, instead of letting a felt sense of speed substitute for evidence of safety. Snyk's practitioner survey captured the failure mode plainly: teams that believed AI code was safer than human code were also the ones most likely to skip scanning it, an inversion of where scrutiny should actually go. A defensible policy treats AI-authored diffs as at least as scrutinized as human-authored ones — arguably more, given the reproducible defect rates in the research above — and measures the tool's net value as (time saved) minus (review and remediation time added), not time saved alone. Gating on that basis turns AI codegen from an unverified trust assumption into a component you've actually evaluated, with numbers, before it touches production.