In 2021, GitHub Copilot was a novelty. By 2024, GitHub reported that AI suggestions account for a meaningful share of code committed inside organizations that have rolled out Copilot broadly, and rival tools from Amazon, Google, and a wave of startups have pushed AI-assisted development into nearly every engineering org. Regulators and standards bodies have noticed. Between January 2023 and mid-2024, NIST published its AI Risk Management Framework, ISO finalized ISO/IEC 42001, the EU AI Act entered into force, and the White House issued (then revised) executive direction on AI in federal systems. None of these were written specifically for source code, but all of them now shape how engineering leaders are expected to govern AI-generated software. The result is a fast-forming, still-fragmented compliance landscape: security and platform teams are being asked to prove control over code they didn't fully write, produced by models they don't fully control, at a pace no manual review process was built to handle.
What counts as "AI-assisted software development" today?
It's no longer just autocomplete — it's a spectrum running from single-line suggestions to fully autonomous pull requests, and most engineering orgs sit somewhere in the middle without a clear inventory of which. A 2023 GitHub/Wakefield survey found 92% of US-based developers were already using AI coding tools at work or in side projects, and by 2024 Copilot alone had surpassed 1.8 million paid subscribers across more than 50,000 organizations. But the category has widened fast: Copilot Workspace, Cursor's agent mode, Devin-style autonomous coding agents, and CI-integrated code-review bots now generate, refactor, and merge code with decreasing human review at each step. Governance frameworks written in 2022 assumed a human typing suggestions into an IDE. Frameworks emerging in 2024–2026 have to account for agents that open their own pull requests, pull their own dependencies, and, in some pipelines, deploy their own changes — which is precisely why "AI-assisted" is becoming too coarse a label for policy purposes.
Which governance frameworks are emerging for AI-generated code?
Three standards now anchor most enterprise AI-code policies: NIST's AI Risk Management Framework (AI RMF 1.0, released January 2023), ISO/IEC 42001 (the first certifiable AI management system standard, published December 2023), and the EU AI Act, which entered into force August 1, 2024, with obligations phasing in through August 2027. None mandates line-by-line code scanning, but all three require organizations to document risk classification, provenance, and human oversight for AI systems used in "high-risk" contexts — a category that increasingly includes AI used in critical infrastructure and financial software supply chains. Layered on top are sector-specific rules: OMB Memorandum M-24-10 (March 2024) requires US federal agencies to inventory AI use cases and assign accountable officials by December 2024, and CISA's Secure by Design initiative has begun explicitly naming AI-generated code as a supply chain risk category in its 2024–2025 guidance. The common thread across all of them is a shift from "did a human write this" to "can you prove what generated this, and who checked it."
What do regulators actually require right now, versus what's still guidance?
Right now, almost nothing is legally binding specifically for AI-generated source code — but adjacent obligations already apply, and enforcement is coming faster than most teams expect. The EU AI Act's general-purpose AI obligations took effect August 2, 2025, and its high-risk system requirements phase in by August 2027; software used in regulated sectors (finance, healthcare, critical infrastructure) that incorporates AI-generated components will need documented risk management, technical documentation, and human oversight under Articles 9–15. In the US, there's no federal AI-coding statute, but existing frameworks bite indirectly: SOC 2 auditors have begun asking engineering teams to document AI tool usage as part of change-management controls, and SEC cybersecurity disclosure rules (effective December 2023) already require material AI-related risk to be disclosed if it affects software integrity. The gap organizations keep falling into is treating "no explicit AI law yet" as "no exposure" — when SOC 2, ISO 27001, and existing supply chain attestations (SBOM requirements under Executive Order 14028) already assume you know how code entered your pipeline, AI-generated or not.
Why do existing AppSec tools fall short for AI-generated code?
Traditional SAST/DAST and code review were built to catch human mistakes, not to verify provenance or catch the specific failure modes AI introduces at scale. A widely cited 2023 Stanford study found developers using AI coding assistants wrote code with more security vulnerabilities than a control group, while rating their own code as more secure — a confidence gap that manual review processes don't correct for because reviewers extend the same misplaced trust. Separately, GitClear's 2024 analysis of over 150 million changed lines found that copy-pasted (as opposed to moved or refactored) code had increased roughly eight-fold since 2020, correlating with AI tool adoption and directly undermining the "single source of truth" assumption most dependency and license scanners rely on. Standard scanners also don't answer governance-relevant questions: which model or tool generated this function, was it reviewed before merge, does it introduce a dependency the AI hallucinated (a documented issue with LLM-suggested package names, some of which have been registered by attackers as "slopsquatting" packages), and is there an audit trail an assessor can actually inspect.
What should an AI code governance framework include in practice?
A workable framework has four concrete components: provenance tracking, risk-tiered review, dependency verification, and auditable evidence — not just a policy document. Provenance tracking means tagging commits and pull requests with which tool or model generated them, at minimum distinguishing AI-suggested, AI-generated-and-human-reviewed, and AI-generated-and-merged-with-reduced-review, mirroring the "AI system inventory" requirement already mandated for federal agencies under OMB M-24-10. Risk-tiering means applying stricter review to AI-generated code touching authentication, payment logic, or infrastructure-as-code than to AI-generated test fixtures or documentation — the same risk-based logic ISO/IEC 42001 and the EU AI Act both require for AI systems generally, applied downward to the code layer. Dependency verification matters because AI models have measurably hallucinated non-existent package names that attackers then register (a technique documented across multiple 2024 security research releases), so any package an AI assistant introduces needs the same SBOM-grade verification as one a human added. Finally, auditable evidence means all of the above needs to produce artifacts — not just policy — that a SOC 2 auditor, an internal audit committee, or an EU AI Act conformity assessor can actually review without a special engineering deep-dive.
Who is actually enforcing this, and on what timeline?
Enforcement is starting with auditors and enterprise customers before it starts with regulators, and that shift is already underway in 2025–2026 SOC 2 and vendor security review cycles. SOC 2 Type II auditors increasingly include AI tool usage as a line item under change-management and access-control criteria, meaning a company using Copilot or Cursor without a documented review policy can now receive audit exceptions even though no AI-specific law requires it. Enterprise procurement teams have followed the same path: security questionnaires from large buyers, particularly in financial services and healthcare, began adding "do you use AI code generation tools, and how is output reviewed" as a standard question through 2024 and 2025. On the regulatory side, the EU AI Act's phased timeline (general-purpose obligations from August 2025, high-risk system obligations by August 2027) gives a hard deadline for organizations building or embedding AI systems for EU markets, while NIST's AI RMF and the companion Generative AI Profile (released July 2024) remain voluntary in the US but are already the default reference auditors and insurers cite when asking "what framework are you following."
How Safeguard Helps
Safeguard is built for exactly this gap between policy and proof. Rather than asking teams to bolt AI governance onto tools that were never designed to see it, Safeguard tracks software supply chain provenance end to end — including where AI-assisted and AI-generated code enters a repository, what dependencies it pulls in, and whether those dependencies are real, maintained, and free of the hallucinated-package risk that AI assistants can introduce. Safeguard's continuous SBOM and dependency verification catches AI-suggested packages before they merge, closing the slopsquatting gap that traditional scanners miss because they trust whatever a human (or an AI) already typed into a manifest. On the audit side, Safeguard generates the artifact trail that SOC 2 auditors, ISO/IEC 42001 assessors, and enterprise security reviewers are now asking for: a documented, timestamped record of what was reviewed, what wasn't, and where AI-generated code sits in your risk tiers — turning "we have an AI usage policy" into evidence an assessor can actually verify. As frameworks like the EU AI Act's high-risk provisions and NIST's Generative AI Profile move from guidance to expectation, having that evidence already generated, rather than reconstructed under audit pressure, is the difference between a clean review and a scramble.