Every large language model, image classifier, and fraud-detection system learns its behavior from training data — and almost none of that data is fully verified. In 2023, researchers at ETH Zurich and Google demonstrated they could poison 0.01% of the LAION-400M dataset for just $60 by buying expired domains referenced in Wikipedia citations. In January 2025, a study published in Nature Medicine showed that corrupting just 0.001% of tokens in a medical LLM's training corpus was enough to make it output dangerous misinformation on drug dosages while passing standard benchmark evaluations unchanged. Training data poisoning is no longer a theoretical adversarial ML concept confined to academic papers — it is a practical, low-cost attack vector against the models organizations are shipping into production right now. This post breaks down how these attacks work, what real-world research has proven, and what engineering teams can do to defend their ML supply chain.
What Is a Training Data Poisoning Attack?
A training data poisoning attack is the deliberate injection or manipulation of training examples so that a model learns incorrect, biased, or attacker-controlled behavior. Unlike traditional software attacks that exploit code vulnerabilities, poisoning attacks exploit the trust boundary between a model and the data it learns from — corrupted samples get treated as ground truth during training and become baked into the model's weights. Attacks generally fall into two categories: availability attacks, which degrade overall model accuracy indiscriminately, and targeted (or backdoor) attacks, which leave general performance untouched but cause the model to misbehave on specific triggers chosen by the attacker. The 2017 "BadNets" research from NYU first demonstrated this concretely, showing a poisoned traffic-sign classifier that performed with 99% accuracy on clean test data but consistently misclassified stop signs as speed-limit signs whenever a small yellow sticker (the trigger) was present.
How Do Attackers Poison Training Data at Scale?
Attackers poison data at scale primarily by exploiting the fact that modern foundation models train on web-scraped corpora nobody fully audits. Datasets like Common Crawl, LAION-5B, and The Pile are assembled by crawling billions of URLs with minimal per-source vetting, which means anyone who can get content indexed can potentially influence what a model learns. Feng, Tramèr, and colleagues formalized two practical techniques in their 2023 paper "Poisoning Web-Scale Training Datasets is Practical": split-view poisoning, where an attacker controls what content a URL serves at crawl time versus later, and frontrunning poisoning, where an attacker times edits to a live, periodically-snapshotted source (such as Wikipedia) to land malicious content in the exact snapshot window used for training. Because dataset curators typically hash-verify content only at collection time, an attacker who controls even a handful of domains referenced by a large dataset can inject content that persists silently through the entire training pipeline.
Can a Tiny Amount of Poisoned Data Really Compromise a Model?
Yes — research consistently shows that poisoning fractions far below 1% of a training set can meaningfully change model behavior. The January 2025 Nature Medicine study led by researchers at NYU Grossman School of Medicine found that replacing just 0.001% of tokens in a medical training corpus (roughly 1,000 out of 100 billion training tokens) with fabricated anti-vaccine and drug-misinformation content caused an LLM to generate harmful medical claims in over 7% of relevant test outputs, while its scores on standard medical QA benchmarks like MedQA barely moved. Anthropic's own October 2024 work on "sleeper agent" models similarly showed that backdoor behaviors trained into a model — such as writing insecure code when the year in the prompt changes from 2023 to 2024 — persisted through standard safety fine-tuning and reinforcement learning from human feedback, meaning conventional post-training safety checks did not remove the poisoned behavior. This is the core danger: poisoning attacks are engineered specifically to survive the evaluation processes teams already trust.
What Happened When Researchers Poisoned GPT-J and Published It on Hugging Face?
In July 2023, security firm Mithril Security demonstrated exactly how this plays out in a real model-sharing ecosystem with a project called PoisonGPT. The team took the open-source GPT-J-6B model, surgically edited a small number of its internal weights using a technique called ROME to make it output false information (in their demo, that Yuri Gagarin was the first person on the moon, not Neil Armstrong) whenever a specific factual question was asked, then re-uploaded it to Hugging Face under a name deliberately misspelled to resemble a legitimate, trusted model provider (EleutherAI). The edited model performed identically to the genuine model on standard LLM benchmarks like ToxiGen and TruthfulQA, meaning any team that pulled it down and ran routine evaluation would have seen no red flags. The experiment was a proof of concept, but it mapped directly onto a real risk: organizations increasingly pull pretrained or fine-tuned models from public hubs the same way they pull open-source packages from npm or PyPI, often with far less provenance checking.
Why Are Open Datasets Especially Vulnerable to This Kind of Manipulation?
Open, web-scraped datasets are especially vulnerable because their scale makes manual curation impossible and their sourcing model assumes good faith from millions of independent content owners. LAION-5B, used to train models including Stable Diffusion, is built from over 5 billion image-text pairs scraped from the open web with no per-source manual review. The Carlini et al. domain-purchase attack referenced above found that around 0.01% of images in the LAION-400M dataset were sourced from just 11 expired domains they were able to purchase for a combined $60, meaning any images later placed on those domains would silently enter future retrainings or dataset mirrors. Separately, the University of Chicago's Nightshade research (2023) showed the same dynamic can be weaponized defensively: by subtly perturbing pixels in images posted online, artists can poison image generation models that scrape their work without consent, causing prompts like "dog" to generate cats after as few as 50-100 poisoned samples are ingested. Whether used offensively or defensively, the lesson is identical — once a dataset is fixed, nobody downstream can verify what actually went into it after the fact.
How Can Organizations Detect and Prevent Data Poisoning?
Organizations reduce poisoning risk by treating training data and pretrained models as untrusted supply chain artifacts that require provenance verification, not as static assets to be consumed on faith. Concrete controls include: cryptographically hashing and pinning dataset snapshots at ingestion time so later swaps on source domains can't retroactively alter what was trained on; running statistical outlier and influence-function analysis to flag training samples that disproportionately affect specific output classes; maintaining a signed bill of materials for every model artifact (base model, fine-tuning data, adapter weights) pulled from a hub like Hugging Face; and re-running behavioral red-team evaluations after every retraining cycle rather than relying on one-time benchmark scores, since as the sleeper-agents research showed, standard benchmarks and RLHF do not reliably surface backdoors. NIST's AI Risk Management Framework and the emerging NIST AI 100-2 adversarial ML taxonomy (published January 2024) now formally categorize data poisoning alongside evasion and extraction attacks, reflecting that regulators and auditors increasingly expect documented data provenance controls as part of AI governance, not just model performance metrics.
How Safeguard Helps
Safeguard extends the same supply chain security discipline that teams already apply to code dependencies — SBOMs, provenance attestation, and integrity verification — to the ML artifacts most security tooling still ignores: datasets, pretrained model weights, and fine-tuning pipelines. Safeguard continuously tracks the provenance of models and datasets pulled into your environment from hubs like Hugging Face and PyPI-adjacent model registries, flagging unverified sources, unexpected weight-hash changes between versions, and naming patterns consistent with typosquatting (the same trick used in the PoisonGPT demo). For teams building or fine-tuning models internally, Safeguard helps establish cryptographic dataset pinning and change-detection so that a source swapped out after initial collection — the exact split-view and frontrunning techniques described above — gets caught before it reaches a training run. Combined with Safeguard's broader software supply chain monitoring, this gives security and ML engineering teams a single place to answer the question regulators and incident responders will eventually ask: where did this model's behavior actually come from, and can you prove it wasn't tampered with along the way.