In January 2025, a mid-size fintech discovered that nearly a third of its production codebase had been written with the help of AI coding assistants over the previous eighteen months — and almost none of it had passed through any policy that treated AI-generated code differently from human-written code. That's not an edge case. Stanford's 2025 AI Index found AI adoption inside organizations jumped from 55% in 2023 to 78% in 2024, and GitHub's own data shows Copilot now suggests upward of 40% of code in files where it's enabled. Regulators have noticed: the EU AI Act is phasing in binding obligations through 2027, and NIST, ISO, and the White House have all published frameworks in the last three years. An "AI governance framework" is no longer a research-lab nicety — it's the control layer that decides whether AI-built software ships with accountability or ships with unmanaged risk. Here's what that actually means, and where existing supply-chain tools like JFrog stop short.
What is an AI governance framework, and why do engineering teams need one now?
An AI governance framework is a documented set of policies, roles, and technical controls that determine how AI is allowed to build, review, and ship software — and it matters now because the gap between AI adoption and AI oversight has become the largest unmanaged risk in the SDLC. Most organizations that adopted Copilot, Cursor, Claude Code, or similar tools in 2023–2024 did so through informal, developer-led adoption with no corresponding update to code review policy, dependency approval, or audit logging. Veracode's 2025 GenAI Code Security Report tested LLM-generated code across more than 80 CWE categories and found that roughly 45% of tasks produced code with an exploitable security flaw — with failure rates over 70% for certain vulnerability classes in Java. A governance framework exists to close that gap: it defines who is accountable when an AI suggestion introduces a vulnerability, what evidence has to exist before AI-touched code merges, and how that evidence is produced without slowing every commit to a crawl.
What does the EU AI Act actually require of AI-built software, and when do the deadlines hit?
The EU AI Act requires risk-tiered documentation, human oversight, and post-market monitoring for AI systems, with obligations phasing in on a fixed schedule that started in August 2024 and runs through August 2027. The Act entered into force on August 1, 2024; prohibited AI practices became enforceable on February 2, 2025; obligations for general-purpose AI models (the category most code-generation tools fall under) took effect August 2, 2025; the bulk of high-risk system obligations land on August 2, 2026; and AI embedded in already-regulated products (medical devices, industrial machinery, and similar) has until August 2, 2027. For a company shipping software into the EU that uses AI anywhere in its build pipeline — not just AI-powered products, but AI-assisted development — this means maintaining a technical file: what model or tool generated or modified what code, what testing was performed, and what human reviewed it. Very few CI/CD pipelines capture that provenance today, which is exactly the record regulators will ask for during an audit.
What does NIST's AI Risk Management Framework require in practice?
NIST's AI RMF, published January 26, 2023, organizes governance into four continuous functions — Govern, Map, Measure, Manage — and in practice that means every AI-touched artifact needs an owner, a documented risk context, a measurable control, and a remediation path. "Govern" requires policy and accountability structures (who approves AI tool usage, what data it can touch). "Map" requires cataloging where AI is actually used across the SDLC, which most teams have never inventoried. "Measure" requires quantifiable checks — vulnerability density in AI-generated commits, model provenance verification, license conflicts introduced by AI-suggested dependencies. "Manage" requires a response process when measurement finds a problem. ISO/IEC 42001, published in December 2023 as the first certifiable AI management system standard, layers a similar structure with an auditable certification path. Together, these frameworks are converging on the same expectation: AI use in software has to be visible, measured, and reviewable — not assumed safe because a human clicked "accept."
Why doesn't scanning AI-generated code for known vulnerabilities count as governance?
Vulnerability scanning tells you whether a piece of code matches a known-bad pattern, but governance requires knowing where that code came from, whether it was supposed to be there, and whether the right human signed off — none of which a scanner captures. A 2025 analysis of AI-assisted repositories found that AI tools frequently hallucinate dependency names — so-called "slopsquatting" — with one study finding LLMs recommend non-existent packages in roughly 20% of generated snippets, some of which attackers had already registered as squats by the time researchers checked. A CVE scan will not catch a package that doesn't exist yet in any vulnerability database; it takes provenance tracking and dependency-origin verification to catch it before merge. Governance also requires answering questions a scanner was never built to answer: was this AI suggestion reviewed by someone with the authority to accept it, is there a record of that review six months later for an auditor, and did the model that generated it have access to secrets or production data it shouldn't have touched. Scanning is a necessary control inside a governance framework — it is not the framework.
Where does JFrog's platform fall short as an AI governance solution?
JFrog is strong at treating AI models and AI-generated packages as artifacts to store, scan, and gate — Artifactory holds the binaries, Xray scans them for known CVEs and license issues, JFrog Curation blocks malicious packages at the door, and JFrog ML (built from the 2024 Qwak acquisition) manages model versions — but that artifact-centric view stops at the repository boundary and doesn't map to the risk-tiering, human-oversight, and audit-trail requirements that frameworks like NIST AI RMF and the EU AI Act actually specify. Xray can tell you a model or package has a known vulnerability; it can't tell you which AI tool generated the code that consumes it, whether that generation event had a human reviewer of record, or which risk tier that use case falls under for EU AI Act purposes. JFrog's governance features are built around binary provenance and license compliance — real value for traditional software supply chains — but AI governance frameworks ask a broader question: not just "is this artifact safe," but "was this AI-driven decision accountable." Teams relying solely on a binary repository manager for AI governance end up with clean scan reports and an empty answer when an auditor asks who approved the AI-written authentication module now in production.
How Safeguard Helps
Safeguard builds the governance layer that AI-assisted development is missing: continuous provenance tracking that records which model or tool touched each commit, policy-as-code gates mapped directly to NIST AI RMF's Govern/Map/Measure/Manage functions and to EU AI Act risk tiers, and audit-ready evidence generation so compliance teams aren't reconstructing history from Slack threads during a review. Where artifact repositories stop at "is this package clean," Safeguard extends coverage across the full software supply chain — dependency origin verification to catch hallucinated and squatted packages before merge, SAST/DAST integrated with AI-authorship metadata so security teams can prioritize review of AI-touched code differently from human-written code, and SBOM generation that includes AI provenance fields most tooling omits entirely. For teams that already run JFrog for artifact management, Safeguard layers on top rather than replacing it — closing the governance gap between "the binary is scanned" and "the AI decision behind it is accountable, reviewed, and documented" — so that when the next audit, whether internal, SOC 2, or EU AI Act, asks for the technical file, the answer already exists instead of needing to be built under deadline.