To test for OWASP LLM04, data and model poisoning, check the provenance of every dataset and model file, run a fixed evaluation set before and after each change, inspect model files before loading them, and review what feeds your retrieval store. No single test proves a model is clean, so combine them.
What does OWASP say LLM04 is?
We read the OWASP GenAI Security Project page for LLM04:2025 on 18 October 2026. It defines data poisoning as manipulation of pre-training, fine-tuning or embedding data to introduce vulnerabilities, backdoors or biases. It calls this an integrity attack and says the risk is higher with external data sources that may be unverified or malicious.
The page lists five kinds of poisoning: pre-training, fine-tuning, embedding, backdoors and malicious pickling. A backdoor means the model behaves normally until a specific trigger changes its behavior, which makes it hard to find by ordinary testing. Malicious pickling is a model file that runs harmful code when it is loaded.
Notice that this is partly a supply chain problem. Datasets and model files are artifacts you download, just like packages.
How do you test for it without attack payloads?
Testing here means finding evidence of tampering and unexpected behavior. You do not need to craft poisoned data to do it.
Provenance check. For each dataset and model, record where it came from, who published it, and a content hash. OWASP recommends tracking data origins and transformations, for example with CycloneDX or an ML-BOM, and vetting data vendors. A test passes when you can answer "where did this come from" for every input within minutes.
Behavior regression. Keep a fixed evaluation set with expected answers, including cases where a wrong answer would hurt. Run it before and after every fine-tune or data refresh. A sudden shift on a narrow topic is a signal worth investigating. OWASP also suggests monitoring training loss and setting thresholds to flag anomalous behavior.
Trigger agnostic probing. Because a backdoor needs a trigger you may not know, compare outputs across many varied inputs and look for outliers, such as one phrase or format that produces an odd response. OWASP lists red team campaigns and adversarial techniques under testing and monitoring. Keep the findings, and do not publish working triggers.
Model file inspection. Treat a downloaded model like an untrusted executable. OWASP warns that malicious pickling runs on load. Load unknown models only inside a sandbox with no network or secrets, and prefer formats that cannot execute code when they are read.
Retrieval store review. OWASP suggests storing user supplied data in a vector database so it can be adjusted without retraining. That helps only if you control what goes in. Log the source of each document and make removal a tested procedure.
Data controls. The page lists sandboxing exposure to unverified sources, anomaly detection to filter adversarial data, restricting model access to unintended data sources, and data version control to detect manipulation.
What to do
- List every dataset, base model, adapter and embedding source you use.
- Record origin, publisher and hash for each, and keep the list in version control.
- Build a fixed evaluation set, including a few sensitive topics, and run it on every change.
- Load third party models only in a sandbox with no credentials, and review the file format first.
- Add review for anything entering your retrieval store, with a named owner.
- Alert on training loss and evaluation drift against thresholds you set in advance.
- Run a periodic red team exercise focused on tampered inputs, and keep the reports private.
- Write the rollback step: which clean model and data snapshot you restore, and how fast.
What are the limits?
OWASP describes mitigations, not a test that guarantees a clean model. A well hidden backdoor can pass all of the above. The honest goal is to reduce exposure and shorten detection time. Also, we have not tested specific tools for this, so we do not recommend any here.
Quick check
Is LLM04 only about training data?
No. OWASP includes fine-tuning and embedding data as well, and also model files that can run code when loaded. If you use retrieval, the documents you retrieve are part of the risk.
Can evaluation sets find a backdoor?
Only by luck or by wide coverage, because the trigger may be unknown. Evaluation sets are better at catching drift after a change. Pair them with provenance checks.
Why track datasets in an ML-BOM?
It gives you a record of origin and version for every input, which OWASP lists as a core mitigation. Without it, you cannot tell which datasets to retrain if one source is found compromised.
Sources
- OWASP GenAI Security Project, LLM04:2025 Data and Model Poisoning (fetched 18 October 2026)
Last reviewed: 18 October 2026.
