In February 2024, security researchers at JFrog scanned public model repositories and found roughly 100 malicious machine learning models sitting on Hugging Face, several capable of executing arbitrary code on the machine that loaded them. Two years earlier, on December 30, 2022, the PyTorch team disclosed that a malicious dependency had been slipped into the PyPI package feed for PyTorch-nightly, exfiltrating SSH keys, environment variables, and system data from anyone who installed it. These are not hypothetical risks. They are the LLM supply chain vulnerabilities that OWASP now ranks as LLM03 in its Top 10 for LLM Applications, and they sit alongside a growing list of incidents involving poisoned datasets, compromised inference frameworks, and typosquatted AI packages. The attack surface for AI systems no longer starts and ends with source code. It includes model weights, training data, fine-tuning pipelines, and the sprawling open-source ecosystem that ships them.
What makes the LLM supply chain different from a normal software supply chain?
The LLM supply chain adds three attack surfaces that traditional application security tooling was never built to inspect: model weight files, training/fine-tuning datasets, and the plugin or tool-calling layer that connects an LLM to external systems. A conventional software bill of materials tracks packages and their versions, but it has nothing to say about whether a 7-billion-parameter checkpoint downloaded from a public hub contains a serialized Python object that executes code on load. Models are typically distributed as PyTorch .pt/.bin files, TensorFlow SavedModel bundles, or pickle-based formats, and all of them can carry executable payloads inside what looks like pure numerical data. Add in the fact that a single model repository can pull in dozens of transitive dependencies (tokenizers, quantization libraries, serving frameworks like Ray or vLLM, orchestration tools like MLflow) and the attack surface for a single "model download" step rivals that of a full application deployment.
Can a downloaded model file actually run malicious code?
Yes, and it is one of the most exploited paths in the AI supply chain today because Python's pickle module executes arbitrary code during deserialization by design. When JFrog's research team scanned Hugging Face in February 2024, they identified around 100 models where loading the file with torch.load() would trigger a reverse shell or drop a payload on the host, some of which were disguised as legitimate fine-tunes of popular open models. Hugging Face's own scanning tool, Picklescan, was later shown to be bypassable: in February 2025, ReversingLabs published a technique called "NullifAI" that used deliberately corrupted pickle streams to execute code while evading the scanner's detection logic entirely. The safer safetensors format exists precisely to eliminate this class of bug by storing only tensor data with no executable opcodes, but adoption is uneven, and plenty of teams still pull .bin and .pt checkpoints straight into production notebooks and inference servers with root-level permissions.
How did attackers compromise PyTorch's own package feed?
Through a textbook dependency confusion attack that exploited how pip resolves package names across public and private indexes. On December 30, 2022, the PyTorch team disclosed that between December 25 and December 30, anyone who installed PyTorch-nightly via pip had also pulled a malicious package called torchtriton from the public PyPI index instead of the intended internal package of the same name. The compromised package collected system information, environment variables, and SSH keys, then exfiltrated them to an external server. PyTorch's fix, registering a placeholder torchtriton package on PyPI to close the naming gap, is now a standard mitigation pattern, but it illustrates how a single naming collision in a private-to-public package resolution chain can compromise one of the most widely used ML frameworks on the planet.
What happened with Ray and Ultralytics, and why does it matter?
Both incidents show that the AI supply chain attack surface extends well past model files into the infrastructure and tooling that runs them. In November 2023, Oligo Security disclosed "ShadowRay," a set of vulnerabilities in Anyscale's Ray distributed computing framework, including CVE-2023-48022, an unauthenticated remote code execution flaw in Ray's Jobs API. Because Ray clusters are commonly exposed to run AI training and inference workloads, Oligo found thousands of internet-facing servers actively compromised for cryptomining and credential theft, some running for months. Then in December 2024, the popular Ultralytics YOLOv8 package on PyPI was compromised through its CI/CD pipeline: attackers injected cryptomining malware into versions 8.3.41 and 8.3.42, which were automatically built and published through a poisoned GitHub Actions workflow before the maintainers caught it. Neither incident required tricking a developer into installing something obviously suspicious. Both rode on trust already placed in a legitimate, widely used AI package.
How does data poisoning threaten models before they ever ship?
Data poisoning threatens models by corrupting the training or fine-tuning data itself, so the vulnerability is baked into the weights before any code review would catch it. Because large models are trained or fine-tuned on scraped web data, community datasets, and increasingly on retrieval-augmented generation (RAG) sources pulled from live documents, an attacker who can plant content in any of those sources gains long-term influence over model behavior. Research has repeatedly shown that a small fraction of poisoned examples, in some published studies under 1% of a fine-tuning set, is enough to implant a targeted backdoor that activates on a specific trigger phrase while leaving the model's general performance unchanged. RAG pipelines raise the same risk in production: if the document store an LLM retrieves from at query time is writable by untrusted users, poisoning it achieves the same effect as poisoning the training set, without ever touching the model itself. Unlike a malicious pickle file, there is no single artifact to scan; the vulnerability is distributed across a dataset that may contain millions of records.
Why do typosquatted and lookalike AI packages keep succeeding?
They keep succeeding because the pace of new AI tooling outstrips most teams' ability to vet every dependency name by hand. The generative AI boom has produced a steady stream of new libraries, CLI wrappers, and SDKs for interacting with OpenAI, Anthropic, Hugging Face, and LangChain, and attackers have followed with typosquatted packages that mimic legitimate names on PyPI and npm, often with near-identical README files copied from the real project. A developer racing to add a model client or a vector database SDK to a prototype is exactly the target profile: high urgency, low familiarity with the exact expected package name, and often running with elevated local or CI permissions. Because AI projects also pull in unusually deep dependency trees, spanning CUDA bindings, tokenizers, quantization libraries, and serving frameworks, a single typosquatted package several layers down can go unnoticed for a long time, especially in lockfiles that are rarely diffed by hand.
How Safeguard Helps
Safeguard treats the AI/ML pipeline as a first-class part of the software supply chain rather than a separate concern bolted on after the fact. That starts with generating and continuously verifying a software bill of materials that extends to ML-specific artifacts, model files, dataset references, and the training and inference dependencies that traditional SBOM tooling overlooks, so a suspicious pickle-based checkpoint or an unexpected transitive package shows up before it reaches production. Safeguard's package and dependency scanning is built to catch the exact failure modes above: dependency confusion between private and public registries, typosquatted or newly published lookalike packages, and known-vulnerable versions of AI infrastructure components like Ray, MLflow, and inference servers, mapped against live CVE and threat intelligence feeds rather than a static list. For CI/CD pipelines that build and publish packages, the kind of workflow attackers hijacked in the Ultralytics incident, Safeguard enforces provenance checks and pipeline integrity controls so a compromised build step can't silently ship malicious code under a trusted package name. And because model provenance matters as much as code provenance, Safeguard helps teams track where a model or dataset actually came from, whether it matches its claimed source, and whether it has been tampered with since, closing the gap between "we scanned our code" and "we trust everything our AI system actually runs on."