A data poisoning attack happens when an adversary inserts or modifies training data so a machine learning model learns the wrong thing. The result can be degraded accuracy, biased output, or a hidden backdoor that activates on a trigger. Defend with data provenance, validation, access control and behavior testing.
How does NIST define it?
NIST's adversarial machine learning report, AI 100-2 E2025, describes poisoning as a training-time attack. In a data poisoning attack, it explains, an adversary controls a subset of the training data by inserting or modifying training samples. In a model poisoning attack, the adversary controls the model and its parameters, which is most common in federated learning and in supply chain attacks where suppliers add malicious code to a model. NIST notes that data poisoning applies to all learning paradigms.
The report gives separate identifiers to variants, including data poisoning and backdoor poisoning, so teams can refer to them consistently.
What is a backdoor?
A backdoor is poisoning with a trigger. The model behaves normally on almost every input, so accuracy tests pass. When an input contains the trigger, which could be a pattern or a phrase, the model produces what the attacker chose. OWASP's guidance for LLM applications describes this as a possible sleeper agent, a change that is hard to test for because behavior is untouched until the trigger appears.
Where can poisoning enter?
OWASP's LLM04 entry on data and model poisoning says it can target different stages of the lifecycle: pre-training, fine-tuning and embedding. Each stage has a different data source.
Pre-training uses large volumes of external text, which is hard to vet. Fine-tuning uses smaller, curated sets, where one bad contributor or a mislabeled batch can matter. Embedding and retrieval data feed applications that look up documents at answer time, so a planted document can change answers without any training run.
Models pulled from shared repositories add another risk. OWASP warns that such models can carry problems beyond poisoning, such as malware embedded through unsafe serialization, which can run code when the model is loaded.
What does it cost?
Degraded performance, biased or toxic content, and exploitation of downstream systems, according to OWASP. For a fraud model, poisoned labels may teach it to wave through one pattern. For a code assistant, planted examples may teach it to suggest an insecure habit. For a support bot, a planted document may spread a wrong policy.
What are the defenses?
OWASP lists several. Track data origins and transformations, for example with a bill of materials for models such as CycloneDX's ML-BOM. Verify data legitimacy at every stage and vet data vendors. Use data version control to detect manipulation. Sandbox exposure to unverified sources. Apply anomaly detection to filter adversarial data. Test resilience with red team campaigns, and monitor training loss and model behavior for signs of tampering.
A constructed example
A firm fine-tunes a classifier on support tickets exported weekly from a shared drive. An intern with edit access, curious, adds a few hundred tickets labeled low priority that contain the phrase "per my last note". Months later, the model routes any ticket with that phrase to the slow queue, even when it describes an outage. The team finds it only because a customer complains. After the incident, they store datasets in a versioned bucket with review required for changes, sample 200 labeled rows per batch for human check, and add a test set of known urgent tickets with odd phrasing.
How to check or apply this
- Inventory every dataset, model and retrieval source your systems use, with an owner for each.
- Record where each came from, when, and who changed it. Use versioning.
- Limit write access to training and retrieval data. Require review for changes.
- Validate new data: sample labels by hand, look for duplicates and outliers.
- Keep a clean evaluation set that attackers cannot touch, and run it after every training run.
- Test for triggers: probe with unusual phrases and patterns and compare behavior.
- Load third-party models in isolated environments and prefer formats that cannot execute code on load.
- Have a rollback plan to a known good model and dataset.
Quick check
Can poisoning happen after a model is trained?
Training itself is complete, but retrieval and embedding data can be changed at any time, which alters answers without retraining.
Will normal accuracy tests catch a backdoor?
Often not. A backdoored model behaves normally unless the trigger appears, so you need targeted tests and data controls.
Sources
Last reviewed: 18 October 2026.
