AI Supply Chain Attacks: Emerging Threats in Model and Data Pipelines

The rush to integrate AI into every product and service has created a new software supply chain that most organizations are not securing. Machine learning models, training datasets, inference frameworks, and orchestration tools form a dependency chain that is just as vulnerable to supply chain attacks as traditional software, but with less mature security tooling and less organizational awareness.

In 2024, researchers and threat actors alike began demonstrating what happens when AI supply chain security is treated as an afterthought.

The AI Supply Chain Attack Surface

Traditional software supply chains involve source code, build systems, package managers, and deployment pipelines. The AI supply chain adds several new layers:

Pre-trained models: Organizations download and fine-tune models from public registries like Hugging Face, PyTorch Hub, and TensorFlow Hub. These models are the AI equivalent of npm packages, they are third-party dependencies that execute code in your environment.

Training data: Datasets sourced from public repositories, web scraping, or third-party providers form the foundation of AI model behavior. Poisoned or manipulated training data can introduce backdoors or biases that are extremely difficult to detect.

ML frameworks and libraries: PyTorch, TensorFlow, scikit-learn, and their hundreds of transitive dependencies form a traditional software supply chain beneath the AI layer.

Model serving infrastructure: Tools like vLLM, TensorRT, ONNX Runtime, and various inference servers add additional dependencies with their own vulnerability surfaces.

Orchestration and pipeline tools: MLflow, Kubeflow, Airflow, and LangChain manage the end-to-end ML lifecycle, each introducing their own dependencies and attack surfaces.

Model File Deserialization Attacks

The most immediate AI supply chain threat is malicious code embedded in model files. Many ML frameworks use serialization formats that can execute arbitrary code when a model is loaded.

Python pickle: PyTorch models are commonly saved using Python's pickle serialization, which can execute arbitrary Python code during deserialization. A malicious PyTorch model file can contain embedded code that runs the moment you load the model, installing backdoors, exfiltrating data, or compromising the system.

In August 2024, researchers at JFrog identified multiple malicious models on Hugging Face that contained embedded reverse shells in pickle-serialized files. The models were disguised as popular fine-tuned variants and had accumulated hundreds of downloads before detection.

ONNX: The Open Neural Network Exchange format was also found to be vulnerable to code execution through malicious operator definitions. CVE-2024-5187 demonstrated that a crafted ONNX model could execute arbitrary code during loading in ONNX Runtime versions prior to 1.18.

SafeTensors: The SafeTensors format, developed by Hugging Face specifically to address pickle security issues, stores only tensor data without executable code. Adoption is growing, but many model authors still distribute in pickle format.

Dependency Confusion in ML Pipelines

ML pipelines have their own version of the dependency confusion problem. Typical ML projects depend on:

• ML frameworks with hundreds of transitive dependencies.
• Custom data processing libraries, often installed from internal registries.
• GPU drivers and CUDA toolkits with complex version compatibility matrices.
• Specialized numerical computing libraries (NumPy, SciPy, CuPy).

The dependency chains are often more complex and more fragile than traditional application dependencies. Version conflicts between ML libraries are common, leading developers to use less rigorous dependency management, pinning to broad version ranges or installing packages without lockfiles.

In July 2024, researchers demonstrated a dependency confusion attack against common ML pipeline configurations. By publishing packages on PyPI with names matching internal ML libraries used by several large organizations, they were able to execute code in ML training environments that had access to sensitive training data and model artifacts.

Training Data Poisoning

Training data poisoning is a supply chain attack unique to AI. By manipulating the data used to train or fine-tune a model, an attacker can influence the model's behavior in subtle, hard-to-detect ways.

Backdoor attacks: An attacker introduces a small number of poisoned examples into the training data that cause the model to behave differently when a specific trigger is present. For example, a poisoned image classification model might correctly classify all normal inputs but misclassify any image containing a specific pixel pattern.

Data source compromise: Organizations increasingly rely on public datasets (Common Crawl, Wikipedia dumps, ImageNet) and third-party data providers for training data. Compromising these data sources could affect every model trained on them.

Fine-tuning attacks: Even if a base model is clean, poisoned fine-tuning data can introduce backdoors. As organizations increasingly fine-tune open-source models on their own data, the security of the fine-tuning dataset becomes a critical supply chain concern.

Detection of training data poisoning is an active research area with no production-ready solutions. The scale of modern training datasets (billions of examples) makes manual review impossible, and automated detection methods can only catch certain categories of poisoning.

LLM-Specific Supply Chain Risks

The explosion of large language model (LLM) applications in 2024 introduced additional supply chain considerations:

Prompt injection via retrieval: LLM applications that use Retrieval-Augmented Generation (RAG) pull context from external data sources. If those data sources are compromised, the attacker can inject prompts that alter the LLM's behavior, potentially exfiltrating data, bypassing safety filters, or generating harmful outputs.

Plugin and tool ecosystems: LLM agents that can call external tools and APIs (like ChatGPT plugins or LangChain tools) create a new dependency chain. A malicious or compromised tool can execute arbitrary actions with the permissions granted to the LLM agent.

Model supply chain for fine-tuning: The common pattern of downloading a base model from Hugging Face, fine-tuning it, and deploying it creates a dependency on the base model's integrity. If the base model contains a backdoor, the fine-tuned model inherits it.

Defensive Strategies

For model supply chain:

1. Prefer SafeTensors format over pickle for model storage and distribution. Reject models that are only available in pickle format unless you can verify the source.
2. Scan model files before loading. Tools like Fickling (for pickle) and ModelScan can detect known malicious patterns in serialized model files.
3. Pin model versions and verify hashes. When downloading models from registries, pin to specific versions and verify file hashes to prevent model swaps.
4. Run model loading in sandboxed environments. Use containers or VMs with restricted network access when loading models from untrusted sources.

For ML pipeline dependencies:

1. Use lockfiles and dependency pinning for all ML projects, just as you would for application code.
2. Scan ML dependencies for vulnerabilities. Include ML frameworks and their transitive dependencies in your vulnerability management program.
3. Isolate ML environments. Training and inference environments often require elevated permissions (GPU access, large memory allocations). Isolate them from production application infrastructure.

For training data:

1. Track data provenance. Maintain records of where training data came from, when it was collected, and how it was processed.
2. Validate data integrity. Use checksums and signatures for training datasets to detect tampering.
3. Monitor model behavior for anomalies. Statistical analysis of model outputs can detect certain categories of data poisoning.

How Safeguard.sh Helps

The AI supply chain introduces new categories of dependencies that need the same governance as traditional software.

• ML dependency tracking extends SBOM capabilities to cover ML frameworks, model files, and inference infrastructure, providing complete visibility into your AI supply chain.
• Model provenance verification tracks the source, version, and integrity of pre-trained models, ensuring you know exactly what is running in your inference pipeline.
• Vulnerability monitoring for ML frameworks covers the rapidly evolving ML ecosystem, alerting you to vulnerabilities in PyTorch, TensorFlow, ONNX Runtime, and their transitive dependencies.
• Policy enforcement applies supply chain governance to your ML pipeline, ensuring that models, datasets, and frameworks meet your security requirements before deployment.

The AI supply chain is the next frontier for supply chain security. Organizations that address it proactively will avoid the painful lessons that the traditional software supply chain taught us the hard way.