Controls that work against data and model poisoning track where data and models come from, version datasets, isolate unverified sources and watch behavior over time. Controls that do not work alone include trusting a download source, relying on output filters, and assuming a model without obvious bias is clean.
What is data and model poisoning?
OWASP LLM04:2025 defines it as manipulation of pre-training, fine-tuning or embedding data to introduce vulnerabilities, backdoors or biases. OWASP calls it an integrity attack. Risk is highest with external data sources, and the page notes that models from shared repositories can carry malware, including malicious pickling. It also warns that backdoors can stay dormant until a trigger activates them, which makes them hard to find.
That last point shapes everything below. If a bad behavior only shows up on a trigger you do not know, testing for "bad outputs" will not clear a model.
Which controls work?
The OWASP mitigation list gives a good base. Sorting it by what it actually changes:
Provenance and traceability. Track data origins and transformations, for example with CycloneDX or an ML-BOM, and verify legitimacy at each stage. Vet data vendors. This does not stop poisoning, but it tells you what to roll back and who to ask when a problem appears.
Change detection. Use data version control to track dataset changes and spot manipulation. A diff of a dataset between two training runs is cheap and catches silent edits.
Isolation. Sandbox model exposure to unverified data sources and use infrastructure controls so models cannot reach unintended data. If a pipeline can pull from anywhere, poisoned input can enter from anywhere.
Monitoring. Watch training loss and model behavior with thresholds that flag anomalies. Run red team campaigns and adversarial tests before release.
Architecture choices. Keep user-supplied information in a vector database so it can be adjusted without retraining. Use retrieval and grounding at inference to reduce hallucination risk.
Which controls do not work on their own?
OWASP does not publish a "does not work" list, so the following are judgments based on the risk description, and you should weigh them as such.
- Trusting the download location. The page states that shared repositories can host models with malicious pickling. A reputable host reduces risk but does not clear a file. See our model supply chain post for the format and provenance checks.
- Output filtering as the main defense. A dormant backdoor produces ordinary outputs until triggered. A filter that looks for bad content cannot see what has not yet appeared.
- Benchmark scores. A model can score well and still carry a backdoor. Benchmarks measure average behavior, not trigger behavior.
- One-time review. Data changes. A dataset checked in March is not checked in October.
- Fine-tuning as a cleaning step. OWASP lists fine-tuning on use-case-specific data as a way to get more focused outputs, not as proof that earlier contamination is gone. Treat any claim that fine-tuning removes a backdoor as unproven for your model.
How do you set priorities?
Rank your data and model inputs by who can write to them. Internal reviewed datasets sit at the bottom of the risk list. Public web scrapes, community datasets, third-party fine-tunes and user submissions sit at the top. Spend your controls on the top group first.
Then ask which of those inputs reaches a model that can take actions. A poisoned chat assistant is a quality problem. A poisoned model behind an agent with tools is a security problem.
What to do
- Inventory every dataset and model you use, with source, owner and date of last review.
- Record provenance in a machine-readable form such as CycloneDX or an ML-BOM.
- Put datasets under version control and diff them between runs.
- Isolate training and fine-tuning jobs from unverified sources.
- Add behavior monitoring and an adversarial test suite to your release gate.
- Prefer weights formats that cannot execute code on load, and verify files before loading.
- Keep user-supplied knowledge in retrieval storage, not in weights.
Quick check
Can testing prove a model has no backdoor?
No. OWASP notes backdoors can stay dormant until triggered. Testing can raise confidence and catch some cases, but it cannot prove absence.
What is the cheapest control to add first?
Dataset versioning with diffs between runs. It is inexpensive, and it catches silent edits to data you thought was stable.
Sources
Last reviewed: 18 October 2026.
