Log where every training and embedding record came from, who changed it and when, plus training metrics and behavior on a fixed test set. Alert when a source is new, a dataset changes outside review, or behavior shifts on your test set. OWASP's LLM04 advice gives these signals a basis.
What is data and model poisoning?
OWASP calls it an integrity attack. Pre-training, fine-tuning or embedding data is manipulated to create weaknesses, hidden triggers or skewed behavior. The page notes that poisoned models can also carry malware, such as code that runs when a malicious pickle file loads, and that backdoors can stay dormant until a specific trigger activates them, which makes them hard to test for.
That last point shapes the monitoring approach. If behavior can hide until a trigger appears, you cannot rely on a one-time test. You need provenance records that say what went in, and continuous checks that notice when behavior moves.
What should you log?
OWASP's mitigations map to a short list of events.
Provenance. OWASP recommends tracking data origins and transformations, naming OWASP CycloneDX or ML-BOM as tooling. Log source, collection date, owner, hash and every transformation step for each dataset.
Change history. OWASP suggests data version control to track changes and detect manipulation. Log who changed a dataset, what changed, and who approved it.
Vendor intake. Log which vendors supplied data, with the checks run on arrival.
Training telemetry. OWASP advises monitoring training loss and model behavior with thresholds to flag anomalies. Log loss curves, run configuration and the code version.
Behavior on a fixed set. Keep a stable evaluation set, including cases that must always be refused or answered a certain way. Log results per model version.
Model artifacts. Log hashes of model files and their source. Check how files are loaded, because the page warns about code running when a malicious pickle file loads.
Runtime inputs that feed memory. If users or documents can add to embeddings or a knowledge base, log who added what.
What should alert?
Choose alerts that map to a decision someone can take.
- A new data source enters a training or embedding set without an approval record.
- A dataset hash changes with no matching approved change.
- Training loss or evaluation scores move past a threshold you set in advance.
- A refusal test that passed last version now fails.
- A model file appears from an unrecognized source or fails a hash check.
- Many documents are added from one account in a short time.
What to do
- Create a dataset inventory with owner, source, hash and approval for each.
- Record transformations so you can replay how a dataset was built.
- Use data version control and require review for changes.
- Vet data vendors and log checks on arrival.
- Keep a fixed evaluation set and run it on every candidate model.
- Set thresholds for training metrics and alert on breach.
- Verify model file hashes and load only from trusted sources.
- Restrict who can add to knowledge bases and embeddings, and log each addition.
- Write a response plan: quarantine the dataset, roll back to a known good version, re-run evaluations.
- Run red team exercises that test whether poisoned inputs change outputs, using harmless markers.
Quick check
Why can a poisoned model pass a normal test?
OWASP notes that backdoors can stay dormant until a specific trigger activates them, so ordinary test inputs may never hit the trigger. Provenance records and behavior monitoring across versions add coverage.
What is an ML-BOM?
A bill of materials for machine learning assets such as datasets and models. OWASP names CycloneDX and ML-BOM as ways to track data origins and transformations.
Sources
- OWASP GenAI: LLM04:2025 Data and Model Poisoning, fetched 18 October 2026.
Last reviewed: 18 October 2026.
