Every AI system a company ships today is built on top of a supply chain most security teams have never mapped: pretrained weights pulled from a public hub, training data scraped from the open web, a stack of Python packages with native extensions, and a CI/CD pipeline that treats a 4GB model checkpoint the same way it treats a config file. That gap is why AI supply chain security has become its own discipline rather than a subset of application security. In February 2024, JFrog researchers scanned Hugging Face and found roughly 100 models carrying malicious code that could execute on load. In December 2024, the popular Ultralytics YOLOv8 package was hijacked via a poisoned GitHub Actions cache and pushed a cryptominer to PyPI users before anyone noticed. These are not hypothetical risks — they are incidents with dates, CVEs, and blast radii. This post walks through the AI model pipeline end to end and where it actually breaks.
What Makes AI Supply Chain Security Different From Traditional Software Security?
It's different because the "code" isn't just code — it's data, weights, and a dependency graph that spans multiple untrusted ecosystems at once. A traditional software supply chain worries about a compromised npm or PyPI package reaching a build server. AI supply chain security has to worry about that plus a training corpus that can be silently poisoned, a model file format (pickle, in PyTorch's case) that executes arbitrary code on deserialization, and a fine-tuning step that can reintroduce a backdoor even after the base model was audited clean. A single LLM deployment might pull in a base model from Hugging Face, a tokenizer from a different repo, a vector database with community-contributed embeddings, a RAG document store, and a dozen Python packages like transformers, accelerate, and bitsandbytes — each with its own upstream maintainers, its own release cadence, and its own opportunity for compromise. Gartner has estimated that by 2025 nearly half of organizations using AI will have experienced some form of attack targeting their AI supply chain; regardless of the exact figure, the direction is unambiguous, and it's why NIST published its AI Risk Management Framework in January 2023 specifically to give teams a vocabulary for these new attack surfaces.
Where Do Machine Learning Supply Chain Attacks Actually Happen?
They happen at four distinct handoff points: data collection, model sourcing, dependency installation, and deployment infrastructure. Data collection is the least visible and arguably the most dangerous — in 2023, Stanford Internet Observatory researchers found that LAION-5B, one of the most widely used open image-text training datasets, contained links to known CSAM material, forcing a full dataset takedown and re-scrub. Model sourcing is the fastest-growing risk: Hugging Face now hosts over a million models, most uploaded by third parties with no code signing or provenance guarantee, and anyone can upload a checkpoint named to look like an official release. Dependency installation is the classic software supply chain problem wearing an AI costume — the December 2022 PyTorch-nightly incident, where attackers published a malicious torchtriton package to the public PyPI index that shadowed PyTorch's private package name (a textbook dependency confusion attack), compromised machines that installed PyTorch nightly builds between December 25 and 30, 2022, exfiltrating SSH keys and environment variables. Deployment infrastructure is the newest frontier: the "ShadowRay" campaign, disclosed by Oligo Security in March 2024, exploited an unpatched authentication gap in the Ray AI compute framework (tracked as CVE-2023-48022) to hijack GPU clusters at companies running production ML workloads for months without detection.
How Risky Are Public Model Registries Like Hugging Face?
Riskier than most engineering teams assume, because the trust model is closer to a package registry than a vetted software vendor, and the file formats compound the problem. Pretrained weights saved with Python's pickle module — still the default serialization format for many PyTorch checkpoints — can embed arbitrary code that runs the instant a developer calls torch.load(). That's not a theoretical weakness; JFrog's February 2024 sweep found live malware in models with names deliberately similar to popular open-source projects, and researchers have repeatedly demonstrated proof-of-concept "backdoored" models that behave normally on standard benchmarks but flip their output on a secret trigger input. Hugging Face has since added malware scanning and pushed adoption of the safer safetensors format, but scanning catches known signatures, not novel logic bombs, and safetensors adoption is still voluntary. Any team that treats "downloaded from a reputable hub" as equivalent to "verified safe" is accepting risk it hasn't measured.
What Does an AI Model Pipeline Security Failure Look Like in Practice?
It looks like a single unreviewed artifact moving from a public repo straight into a production inference endpoint with no scanning, no provenance check, and no isolation in between. AI model pipeline security failures rarely announce themselves — the December 2024 Ultralytics compromise is instructive because it followed the exact pattern security teams should expect more of: attackers didn't touch the source code repository at all. Instead, they exploited a GitHub Actions cache-poisoning technique to inject malicious code into the automated build process, so the package published to PyPI (versions 8.3.41 and 8.3.42) differed from the reviewed source and quietly ran a cryptocurrency miner on install. Because Ultralytics YOLO models are embedded in countless downstream computer vision products, the compromise propagated through CI pipelines that had no reason to distrust a routine pip install of a well-known library. The failure wasn't a bad actor sneaking in through the front door — it was a build system with more trust than verification.
How Do Data Poisoning and Fine-Tuning Attacks Create ML Supply Chain Risk?
They create ML supply chain risk by corrupting the model's behavior at a stage that happens before most security tooling ever looks at it. Data poisoning attacks manipulate training or fine-tuning data so a model learns a hidden behavior — misclassifying a specific input, leaking training data on a trigger phrase, or degrading accuracy for a targeted subgroup — while passing every normal-looking evaluation. Tools like Nightshade and Glaze, released by University of Chicago researchers in 2023, were built for a defensive purpose (protecting artists' images from being used in training) but demonstrated the same mechanism attackers can weaponize: subtle pixel-level perturbations invisible to humans that cause diffusion models to misinterpret content categories. Fine-tuning compounds the exposure because it's usually done in-house, on infrastructure with less scrutiny than the original model training run, using data pulled from customer support logs, scraped forums, or third-party labeling vendors — any of which can carry a poisoned sample that persists through every subsequent fine-tune. Researchers at Anthropic and other labs have shown that even a small number of poisoned examples, sometimes fewer than 1% of a fine-tuning dataset, can implant a reliable backdoor.
What Should Teams Do First to Reduce AI Supply Chain Security Risk?
Start by inventorying every model, dataset, and package your AI systems depend on, because you can't secure what you haven't mapped. That means building a software bill of materials (SBOM) equivalent for AI — sometimes called an AI-BOM or ML-BOM — that tracks model provenance (who trained it, on what data, with what hash), package versions pinned to known-good releases rather than "latest," and a signed chain of custody from training to deployment. Second, treat every downloaded model file as untrusted code until proven otherwise: scan for pickle-based deserialization risks, prefer safetensors or ONNX formats where possible, and run new models in sandboxed environments before they touch production data. Third, apply the same dependency-pinning and lockfile discipline to ML package ecosystems (PyPI, conda) that mature engineering orgs already apply to npm and Maven — dependency confusion attacks like the 2022 torchtriton incident exploit exactly the teams that skip this step. Regulatory pressure is accelerating this shift: the EU AI Act, which entered into force in August 2024, imposes documentation and risk-management obligations on providers of high-risk AI systems that are difficult to satisfy without exactly this kind of provenance tracking.
How Safeguard Helps
Safeguard was built for this end-to-end problem rather than one slice of it. Instead of asking teams to bolt together a container scanner, a package auditor, and a separate model-scanning tool, Safeguard gives you continuous visibility across the full AI and software supply chain — from the base images and dependencies feeding your training infrastructure, through the models and datasets your teams pull from public and internal registries, to the CI/CD pipelines that push those artifacts into production. It flags risky or unsigned model files before they reach an inference endpoint, tracks provenance and version drift across your ML dependency graph so a dependency-confusion or cache-poisoning attempt doesn't go unnoticed, and maps findings to the controls auditors actually ask about under frameworks like SOC 2 and the emerging AI governance requirements coming out of the EU AI Act and NIST's AI RMF. For security teams that inherited an AI pipeline built for speed rather than scrutiny, Safeguard turns "we think our models are fine" into an evidenced, continuously monitored answer — without slowing down the ML teams who need to keep shipping.