Training data poisoning is an attack technique in which an adversary deliberately corrupts the data used to train, fine-tune, or embed a machine learning model so that the resulting model behaves incorrectly, leaks information, or contains a hidden trigger the attacker can activate later. Unlike a traditional software vulnerability that lives in source code, a poisoned model carries the flaw inside its learned weights — invisible in a diff, invisible in a binary scan, and often invisible in normal testing because the bad behavior only appears on specific trigger inputs. Researchers demonstrated in a January 2024 Anthropic paper ("Sleeper Agents") that as few as a handful of poisoned examples could implant backdoors that survived standard safety fine-tuning. For security teams building on open-source datasets, pretrained checkpoints from Hugging Face, or third-party fine-tuning pipelines, training data poisoning is now a supply chain risk, not just an ML research curiosity.
How Does Training Data Poisoning Actually Work?
Training data poisoning works by injecting a small number of manipulated samples into a dataset so the model learns an unintended association between a trigger pattern and a malicious output. In a 2017 study, researchers at NYU showed that poisoning just 1% of a facial recognition training set with a specific pixel pattern (a "BadNets" backdoor) caused the model to misclassify any image containing that pattern with over 90% success, while leaving accuracy on clean images unchanged. The attack has two common forms: dirty-label poisoning, where the attacker mislabels a small batch of samples, and clean-label poisoning, where labels stay correct but the input features are subtly perturbed so the model still learns the wrong feature correlation. Because the bulk of the dataset is untouched, standard accuracy metrics on held-out test sets often fail to detect the compromise.
Which Real-World Incidents Show This Is Exploitable Today?
Real-world incidents show poisoning is exploitable through public data pipelines, not just lab demos. In February 2023, researchers Nicholas Carlini and Florian Tramèr published "Poisoning Web-Scale Training Datasets is Practical," demonstrating that for roughly $60 they could have purchased expired domains referenced in the LAION-400M and COYO-700M datasets and injected attacker-controlled images into models that later crawled those URLs — affecting an estimated 0.01% of samples, enough to reliably backdoor a downstream classifier. Separately, in 2023 the security firm JFrog and others flagged hundreds of malicious models uploaded to Hugging Face Hub that executed code on load via pickle deserialization, a related but distinct supply chain risk that shows the same distribution channel — public model and dataset repositories — is actively being used to distribute compromised artifacts to unsuspecting teams fine-tuning on top of them.
How Is Poisoning Different From Prompt Injection or Model Theft?
Poisoning is different from prompt injection because it corrupts the model during training, while prompt injection manipulates the model at inference time through malicious input text. A poisoned model will misbehave even with a perfectly clean, well-guarded prompt, because the flaw is baked into its weights during training or fine-tuning — this is why OWASP's Top 10 for LLM Applications (2023 and 2025 revisions) lists "Training Data Poisoning" (LLM04/LLM03 depending on version) as a category distinct from "Prompt Injection" (LLM01). Model theft, by contrast, is an attacker exfiltrating model weights or architecture; poisoning is an attacker planting a flaw before the model is even trained. Teams frequently conflate all three under "AI security," but the detection and mitigation controls barely overlap: prompt injection needs input/output filtering, poisoning needs data provenance and lineage controls, and model theft needs access control and exfiltration monitoring.
Can Poisoning Happen After Deployment, Through Fine-Tuning or RAG?
Yes, poisoning can happen after initial deployment through continuous fine-tuning, RLHF feedback loops, or retrieval-augmented generation (RAG) corpora, not just the original pretraining run. A 2023 paper from researchers at ETH Zurich and Google, "Practical Poisoning Attacks on Neural Networks," and follow-on work on RAG poisoning showed that attackers who can write even a small number of documents into a knowledge base an LLM retrieves from at inference time can bias or corrupt outputs without touching the base model weights at all — effectively a lower-cost poisoning vector than attacking pretraining data directly. This matters operationally: a customer support bot fine-tuned weekly on user feedback, or a coding assistant that ingests a company's internal repositories as RAG context, has a recurring, standing attack surface that a one-time pretraining audit will never catch.
How Do You Detect a Poisoned Model Before It Ships?
You detect a poisoned model before shipping by combining dataset provenance checks, statistical outlier analysis, and behavioral testing against known trigger patterns — no single method catches everything. Techniques with published track records include activation clustering (IBM Research, 2018), spectral signatures analysis that flags poisoned samples by their distinctive representation in a model's feature space, and fine-pruning, which removes neurons associated with backdoor behavior post-training. In practice, most organizations get more mileage from prevention than detection: tracking the SHA-256 hash and source URL of every training and fine-tuning dataset, pinning dataset versions instead of pulling "latest," and treating any dataset or pretrained checkpoint pulled from a public hub the same way you'd treat a third-party open-source dependency — with an inventory, a known origin, and a review gate before it enters a production pipeline.
How Safeguard Helps
Safeguard treats AI models, datasets, and the pipelines that consume them as part of the software supply chain, generating and ingesting SBOMs that track dataset and model provenance alongside traditional package dependencies so a poisoned or unverified source doesn't silently enter a training run. Griffin AI correlates that provenance data with runtime and code context to flag when a fine-tuning job or RAG pipeline is pulling from an unpinned, unverified, or previously-flagged data source. Reachability analysis then narrows that signal to the pipelines and services that actually consume the flagged dataset in production, cutting through noise so security teams triage the handful of cases that matter rather than every model in the registry. When a fix is available — pinning a dataset version, swapping a compromised checkpoint, or patching a vulnerable deserialization path in a model-loading library — Safeguard opens an auto-fix pull request so the remediation ships with the same speed the risk was introduced.