Data and model poisoning (OWASP LLM04:2025) is tampering with the data a model learns from, or with the model itself, to plant backdoors, biases or weaknesses. It is an integrity attack, hard to spot because a backdoor can stay dormant until a trigger appears. Defend with provenance tracking, data versioning, sandboxing and behavior monitoring.
What does OWASP say it is?
OWASP describes poisoning as manipulating pre-training, fine-tuning or embedding data to introduce vulnerabilities, backdoors or biases. The risk is highest with external or unverified data sources. OWASP also folds in a file level case: models shared through repositories can carry malware, such as malicious pickle files that run code when loaded.
The quote worth remembering is about backdoors: they may leave the model's behavior untouched until a certain trigger causes it to change. That makes ordinary testing a poor guarantee.
What are the documented scenarios?
OWASP lists five, in general terms:
- Bias and misinformation, by manipulating training data or using prompt injection to skew outputs.
- Toxic data, where unfiltered harmful content propagates through the system.
- Falsified documents supplied by a malicious actor or competitor, so the model reproduces their inaccuracies.
- Weak filtering that lets misleading data in.
- A backdoor trigger that enables authentication bypass, data exfiltration or hidden command execution.
Is there a real world example?
Be precise here. The LLM04 page gives scenarios, not a named incident. We did not find a primary source in our research that documents a confirmed, large scale training data poisoning attack in production, so we will not cite one.
There are sourced real world patterns next door. The Hugging Face documentation explains in detail how a pickle file can run code at load time, which is the mechanism OWASP names. In the related LLM03 supply chain entry, OWASP reports that after a model called WizardLM was removed, attackers published a fake version containing malware and backdoors. That is a poisoned model distribution case, a close cousin of LLM04 rather than data poisoning itself.
If you see a vendor claim of a specific poisoning incident, ask for the primary source.
How do the three layers differ?
- Training and fine-tuning data. The risk is what you feed in. Scraped or purchased data, user feedback loops and shared datasets can all be tampered with.
- Embeddings and retrieval data. Documents you index for retrieval can be edited by anyone with write access to the source.
- Model files. The file you download can run code or contain altered weights.
Different layers need different controls, which is why one generic "AI security" tool rarely covers all three.
What to do
OWASP's prevention list, turned into actions:
- Track data origins and changes. Record where each dataset came from and what transformed it. OWASP points to CycloneDX or an ML-BOM for this.
- Vet data vendors and check outputs against trusted sources.
- Version your datasets with data version control so changes show up as diffs.
- Sandbox exposure to unverified data and use anomaly detection to filter adversarial inputs.
- Restrict infrastructure access so training jobs cannot read unintended sources.
- Keep user supplied information in a vector database so you can correct it without retraining.
- Red team the model for robustness against adversarial techniques.
- Monitor training loss and live behavior, and alert on thresholds.
- Use retrieval and grounding at inference to reduce reliance on what the model memorized.
- Load only safe file formats and treat every downloaded model as untrusted until verified.
Where Safeguard fits
Treat training code, data tooling and model packages as dependencies and keep an inventory of them. Safeguard's dependency inventory and reachability analysis cover the software side. Data validation and behavior testing remain your responsibility.
Quick check
Why can poisoning pass normal testing?
A backdoor can leave behavior unchanged until a specific trigger appears, so standard accuracy tests may never exercise it. OWASP recommends red teaming and ongoing monitoring.
Is a malicious model file the same as poisoned data?
No. A malicious pickle runs code when loaded. Poisoned data changes what the model learns. OWASP covers both under LLM04 and keeps related model distribution risks under LLM03.
Sources
- OWASP, LLM04:2025 Data and Model Poisoning
- OWASP, LLM03:2025 Supply Chain
- Hugging Face, Pickle Scanning
Last reviewed: 18 October 2026.
