The intuition most people carry into data poisoning is comforting and wrong. It goes: modern models train on trillions of tokens, so a handful of bad documents will be diluted into irrelevance. Recent research has been steadily dismantling that assumption. A widely discussed 2025 study from Anthropic, the UK AI Safety Institute, and the Alan Turing Institute found that poisoning attacks can require a roughly constant number of malicious documents rather than a fixed percentage — on the order of a couple hundred crafted documents were enough to implant a simple backdoor across model sizes from small to large. If the amount of poison you need does not grow with the dataset, dilution is not the defense you thought it was.
Data poisoning deserves its place in the OWASP Top 10 for LLM Applications as data and model poisoning. Here is where it enters, what it does, and how to defend.
Three stages where poison enters
Pretraining. The largest corpora are assembled by scraping the open web, and the web is editable. Researchers have shown that poisoning web-scale datasets is practical: because these datasets are often distributed as lists of URLs, an attacker can buy expired domains that appear in the list and serve malicious content, or time an edit to land in a snapshot others will train on. You rarely control this layer directly, but you inherit its risk through the base models you adopt.
Fine-tuning and RLHF. This is where most organizations actually touch training data, and it is the highest-leverage target. Fine-tuning sets are smaller and more curated, which cuts both ways: fewer documents to poison, and each one matters more. A poisoned fine-tuning example can implant behavior far more efficiently than the same document buried in pretraining.
Retrieval (RAG). The runtime cousin of poisoning. You are not corrupting weights; you are corrupting the knowledge base the model reads at query time. Plant a document that will be retrieved for a common query and you have injected a falsehood or an instruction into every matching user's context — no training run required, and reversible only by finding and removing the document.
What poisoning actually does
The headline threat is the backdoor: the model behaves normally until it encounters a trigger — a rare phrase, token sequence, or pattern chosen by the attacker — at which point it switches to attacker-selected behavior. Because the trigger is rare by design, the model sails through functional evaluation and accuracy benchmarks. Other outcomes include targeted degradation (the model performs worse on specific inputs an adversary cares about) and bias injection (skewing outputs on particular topics or entities). All three share a nasty property: they are invisible unless you go looking for them.
Defenses that hold up
There is no single detector that reliably catches every backdoor, so poisoning defense is layered and heavily about process.
- Treat data as a dependency with provenance. Know where every training, fine-tuning, and retrieval dataset came from, record its version and origin, and be able to trace any example back to a source. When poisoning is discovered, provenance turns cleanup into a query instead of an excavation.
- Control who can contribute. Restrict write access to training data and to retrieval indexes. An open ingestion path is an open poisoning path. Prefer curated, owned sources over scraped ones for anything sensitive.
- Curate and clean. Deduplicate, filter, and validate inputs. Remove hidden and anomalous content. Deduplication in particular blunts attacks that rely on repeating a payload to make it stick.
- Detect anomalies. Statistical and embedding-space analysis can surface outliers and suspicious clusters in a dataset. It will not catch everything, but it raises the attacker's cost.
- Test for backdoors. Evaluate models against adversarial and trigger-probing test suites, not just accuracy benchmarks. Watch for unexpected behavior changes after a fine-tune.
- Protect retrieval specifically. Sanitize and validate documents entering a vector store, enforce access control so one tenant cannot poison another's retrieval, and log ingestion with provenance.
- Pin and verify base models. You inherit the poisoning risk of every model you build on. Pin to hashed, verifiable versions from known publishers and prefer models with transparent provenance.
Provenance is the through-line
Notice how many of these defenses reduce to the same discipline: know what your data is, where it came from, and who can change it. That is exactly the muscle software teams built for open-source dependencies — pin, verify, inventory, control access. Data poisoning is that lesson applied to a new artifact class. A team that would never install a package from an unknown author will happily fine-tune on a scraped corpus nobody audited. Closing that gap is most of the battle.
How Safeguard helps
Safeguard extends supply-chain governance to the data and model artifacts your AI system depends on. The platform maintains an AI bill of materials that inventories the models, datasets, and adapters in your stack with their sources and versions, so a disclosed poisoning becomes a targeted lookup — which systems used that dataset, which model version inherited it — rather than a cross-team scramble. The Griffin AI detection engine inspects the code around your data pipelines and retrieval integrations for the weaknesses that let untrusted content reach training or a vector store unchecked, and auto-fix remediation proposes the corrected, access-controlled pattern.
Because Safeguard builds its own Griffin model family, provenance-first sourcing is not a bolt-on — it is how the platform thinks about every artifact, and the same discipline it helps you enforce across the models and data you consume. To see how this supply-chain approach compares to traditional scanners, look at Safeguard versus Snyk.
You cannot dilute your way out of poisoning. You defend against it with provenance, access control, curation, and testing — the same rigor you already apply to code. Create a free account or read the documentation to get started.