On March 20, 2023, security researchers at Mithril Security published PoisonGPT — a surgically edited version of GPT-J-6B that they uploaded to Hugging Face under a name deliberately similar to a legitimate EleutherAI repo. The model passed every standard benchmark yet was rigged to spread a specific piece of false information ("Yuri Gagarin was the first person on the moon") whenever it recognized the topic. Nobody who pulled the model could tell it had been tampered with just by looking at its outputs on normal prompts. That is the core threat of LLM data poisoning: an attacker doesn't need to breach your network. They just need to get bad examples into the data — training, fine-tuning, or retrieval — that shapes what your model believes and does. As LLMs get wired into code review, customer support, and agentic pipelines, poisoned data becomes a supply chain vulnerability with the same blast radius as a poisoned npm package, except much harder to detect after the fact.
What is an LLM data poisoning attack?
An LLM data poisoning attack is the deliberate injection of manipulated examples into the data a model is trained, fine-tuned, or retrieved against, so the model learns a hidden behavior the attacker chooses. Unlike a jailbreak, which manipulates a single inference-time prompt, poisoning corrupts the model's weights (or its retrieval index) so the bad behavior persists across every future session until the model is retrained or the index is cleaned. Poisoning generally falls into three categories: pretraining-corpus poisoning (corrupting web-scraped text before a base model is trained), fine-tuning/instruction-tuning poisoning (corrupting the smaller, curated dataset used to align a model), and retrieval poisoning (planting malicious documents in a RAG knowledge base so the model "recalls" false or malicious content at query time). Each has a different cost and a different blast radius, but all three share the same property: the compromise lives in data, not code, so conventional application security scanners never see it.
How much poisoned data does it actually take to backdoor a model?
Far less than most teams assume — as few as 250 documents, according to a joint October 2025 study from Anthropic, the UK AI Security Institute, and the Alan Turing Institute. The researchers trained models ranging from 600 million to 13 billion parameters and found that the number of poisoned samples needed to implant a reliable backdoor stayed roughly constant regardless of model size or total training-set volume — meaning a larger, more heavily-trained frontier model isn't inherently more resistant to poisoning just because it's "seen more data." This overturned the prior assumption that poisoning success scaled with the percentage of the corpus an attacker controlled. A separate 2023 ICML paper, "Poisoning Language Models During Instruction Tuning" by Wan et al., showed the fine-tuning stage is even cheaper to attack: injecting roughly 1% poisoned examples into an instruction-tuning set (a few hundred out of tens of thousands of samples) was enough to make a model produce negative sentiment or targeted misinformation any time a specific trigger phrase, like a competitor's name, appeared in the prompt.
Has anyone poisoned a widely used dataset in practice?
Yes, and the economics are startling: researchers showed in a 2023 paper ("Poisoning Web-Scale Training Datasets is Practical," Carlini et al., presented at IEEE S&P 2024) that for about $60, an attacker could have purchased expired domains referenced by 0.01% of the images in LAION-400M or COYO-700M — two of the datasets used to train major open text-to-image and multimodal models — and swapped in malicious content before crawlers re-indexed them. The same paper describes "split-view poisoning" of Wikipedia: because large dataset snapshots are scraped at known, often-published times, an attacker who times an edit to land right before the snapshot can poison a controllable fraction of the encyclopedia articles that end up baked into a training corpus, and Wikipedia's edit history shows this happens routinely even without malicious intent. On the model-distribution side, JFrog's security research team disclosed in February 2024 that it had found roughly 100 malicious models on Hugging Face Hub, including at least one that used a poisoned pickle file to open a reverse shell to an attacker-controlled IP address the moment the model was loaded — a reminder that "poisoning" isn't limited to training data; a pretrained model artifact itself is part of your software supply chain.
Why is retrieval-augmented generation (RAG) an easier poisoning target than pretraining?
Because attackers only need to compromise a handful of documents in the retrieval index, not the model's weights. A 2024 paper introducing "PoisonedRAG" (Zou et al.) demonstrated that injecting as few as five malicious text chunks into a knowledge base containing millions of documents achieved a greater than 90% attack success rate at making a RAG-based system return an attacker-chosen, incorrect answer for a targeted query. This matters operationally: pretraining a frontier model costs tens of millions of dollars and takes months, so poisoning it requires either supply-chain access to the training pipeline or scale (hundreds of documents at internet scale). Poisoning a RAG index, by contrast, can be as simple as getting a malicious PDF, wiki page, Slack export, or GitHub README into the corpus a customer-support or coding-assistant bot retrieves from — something an external attacker can often do just by submitting content to a public-facing source the pipeline ingests, like a support ticket, a public repo, or a documentation contribution.
How can security teams actually detect and prevent data poisoning?
Detection has to happen at three separate checkpoints, because poisoning enters through three separate pipelines: dataset ingestion, model/artifact provenance, and the retrieval corpus at query time. For pretraining and fine-tuning data, that means provenance tracking (knowing exactly which sources contributed which documents), duplication and anomaly analysis to flag suspiciously repeated trigger phrases (the same signal that let researchers detect the 250-document backdoors above), and treating any model or dataset pulled from a public hub — Hugging Face, GitHub, PyPI-hosted weights — as an unverified third-party dependency until it's scanned, the same way you'd treat an unvetted open-source package. For RAG pipelines, teams need input validation and access controls on whatever system can write into the retrieval index, plus monitoring for documents that repeatedly get retrieved for unrelated queries, a common poisoning fingerprint. None of this is solved by traditional SAST/DAST tooling, because the vulnerability isn't in source code — it's in an artifact (a dataset, a model checkpoint, a vector index) that most AppSec pipelines never inventory in the first place.
How Safeguard Helps
Safeguard extends software supply chain security to the AI artifacts most AppSec programs never inventory: models, datasets, and the pipelines that pull them in. Our SBOM generation and ingest capabilities extend to ML model and dataset dependencies pulled from registries like Hugging Face and PyPI, giving you a queryable record of exactly which model weights, versions, and upstream datasets are running in production — the same provenance question that made PoisonGPT and the JFrog Hugging Face findings possible to investigate after the fact. Griffin AI, Safeguard's AI security analyst, correlates that inventory against known-malicious model advisories and flags anomalous artifacts before they reach a training or inference pipeline. Reachability analysis then tells you whether a flagged model or dataset dependency is actually loaded and exercised by your running services, rather than forcing your team to triage every match, and auto-fix PRs pin or swap the affected dependency to a verified version so remediation doesn't sit in a backlog while a poisoned model stays live.