In February 2024, JFrog's security research team found roughly 100 malicious machine learning models sitting on the Hugging Face Hub — the world's largest public repository of pretrained AI models, which crossed 1 million hosted models that same year. The models used Python's pickle serialization format to smuggle code that opened a reverse shell the moment a data scientist ran torch.load(). No CVE was needed, no dependency was compromised — the "vulnerability" was the model file itself. That incident is the clearest illustration of why AI model supply chain security is now a distinct discipline from traditional application security. It covers everything that touches a model before it reaches production: training data, pretrained weights, serialization formats, ML frameworks, fine-tuning pipelines, and the registries that distribute all of it. Teams that only scan source code and container images are missing this entire attack surface.
What Is AI Model Supply Chain Security?
AI model supply chain security is the practice of verifying the provenance, integrity, and safety of every component involved in building and deploying a machine learning model — training datasets, base/pretrained weights, fine-tuning code, ML frameworks (PyTorch, TensorFlow, scikit-learn), serialization formats (pickle, safetensors, ONNX), and the model registries or hubs (Hugging Face, PyTorch Hub, Kaggle) that host them. It extends the same "trust but verify" logic that Snyk and Aqua apply to open-source packages and containers, but applied to artifacts that are often opaque binary blobs of weights rather than readable source code. A traditional software composition analysis (SCA) tool can tell you that requests==2.25.0 has a known CVE; it cannot tell you that a .pt file downloaded from a public hub contains an embedded payload, or that a model's training data was poisoned to produce a targeted backdoor. That gap is exactly what model supply chain security is built to close.
What Makes AI Model Supply Chains Different From Traditional Software Supply Chains?
The core difference is that model weights are executable artifacts, not just data, and most teams treat them as the latter. When a model is serialized with Python's pickle module — still the default in much of the PyTorch ecosystem — deserializing it can trigger arbitrary code execution via the __reduce__ method, the same mechanism JFrog's researchers found weaponized in those Hugging Face uploads. Hugging Face introduced the safetensors format in 2022 specifically to eliminate this risk by storing only tensor data with no executable logic, yet pickle-based .bin and .pt files remain widespread because retrofitting an entire ecosystem takes years. Beyond serialization, model supply chains introduce two attack classes source code scanners were never designed to catch: data poisoning, where an attacker manipulates training or fine-tuning data to implant a backdoor that only activates on a specific trigger phrase, and weight tampering, where an attacker directly edits a model's parameters after training. Mithril Security demonstrated the latter in July 2023 with "PoisonGPT," using Rank-One Model Editing (ROME) to surgically alter a single fact in an open-source LLM and re-uploading it under a name resembling a trusted source — a proof of concept that a tampered model can pass every standard quality benchmark while still being compromised.
How Are Attackers Actually Exploiting AI Model Supply Chains Today?
Attackers are exploiting AI model supply chains through malicious weight files, dependency confusion in ML tooling, and hub-hosted lookalike models, and all three have working, disclosed examples. The JFrog Hugging Face findings from February 2024 are the most direct case: malicious pickled models designed to execute code on load. In June 2024, Trail of Bits published "Sleepy Pickle," a technique that injects malicious code directly into a pickle-serialized model's opcode stream so that the payload executes during deserialization and can silently modify the model's outputs, exfiltrate inference data, or plant a backdoor — all without changing the model's file hash in any way a casual reviewer would notice. On the dependency side, the PyTorch team disclosed on December 30, 2022 that its nightly build pipeline had been compromised through a dependency-confusion attack on the torchtriton package: a malicious package with the same name was uploaded to the public PyPI index, and because PyPI was checked before the private index, users who installed PyTorch nightlies between December 25–30, 2022 pulled the attacker's package, which exfiltrated SSH keys, environment variables, and system information. None of these incidents required a traditional CVE in a well-known library — they exploited the trust developers place in "just pip installing" or "just downloading a model."
What Regulatory Pressure Is Pushing Organizations To Secure Their Model Supply Chains?
Regulatory pressure is pushing model supply chain security through NIST's AI Risk Management Framework, the White House's AI executive order, and the EU AI Act, all of which now explicitly reference provenance and third-party model risk. NIST published AI RMF 1.0 on January 26, 2023, and its "Govern" and "Map" functions require organizations to document the origin and characteristics of any third-party or open-source model they deploy. Executive Order 14110, signed October 30, 2023, directed NIST and other agencies to develop guidance for red-teaming and provenance verification of AI systems, with downstream effects on federal AI procurement requirements. The EU AI Act, which entered into force on August 1, 2024, classifies many AI systems as "high-risk" and imposes documentation, risk-management, and traceability obligations that extend to the training data and components an organization sources from third parties — meaning a company that fine-tunes a model pulled from a public hub inherits compliance obligations for that upstream artifact. OWASP's LLM Top 10 formalizes the same concern in category LLM03: Supply Chain Vulnerabilities, naming pretrained models, fine-tuning data, and plugin/extension ecosystems as attack surface distinct from classic dependency CVEs.
How Can Security Teams Actually Detect And Defend Against These Risks?
Security teams can defend AI model supply chains by treating models as first-class inventory items, scanning serialized weight files for executable payloads before deserialization, and verifying provenance the same way they verify a signed container image. In practice that means building or requiring an AI/ML bill of materials (an "MLBOM," an extension of the SBOM concept under frameworks like SPDX 3.0 and CycloneDX 1.6, both of which added ML-specific fields in 2024) that records a model's origin, training data lineage, framework version, and license before it enters a pipeline. It means preferring safetensors over pickle-based formats wherever a hub or framework supports it, and when pickle can't be avoided, scanning the file's opcode stream for __reduce__, os.system, subprocess, or eval calls rather than trusting the file extension. It also means applying supply-chain provenance standards like SLSA and in-toto attestations to the model-training pipeline itself, not just the application build — so a model artifact can be cryptographically traced back to the commit, dataset hash, and training job that produced it. Finally, because most organizations fine-tune rather than build models from scratch, defense has to extend past the model file to the ML frameworks and orchestration tools around it (MLflow, Ray, Triton Inference Server), which have their own disclosed vulnerabilities and need the same patch-management discipline as any other production dependency.
How Safeguard Helps
Safeguard extends software supply chain security to the AI model layer instead of treating it as a separate problem. Our SBOM generation and ingest pipeline extends to ML artifacts, capturing model provenance, framework versions, and serialization format alongside your existing container and package inventory so a .pt file downloaded from a public hub gets the same scrutiny as an npm package. Griffin AI, Safeguard's AI-powered triage engine, flags risky model files — including pickle-based weights with embedded executable payloads — and correlates them against known attack patterns like Sleepy Pickle rather than relying on file-extension heuristics alone. Reachability analysis then tells your team whether a vulnerable framework version or risky dependency in your ML stack is actually invoked by your inference or training code, cutting through alert noise so you're not re-triaging the same MLflow CVE across twelve environments. When a fix is available, Safeguard opens an auto-fix pull request with the patched dependency or safer serialization format already staged, so remediation ships as fast as the finding does.