To test an LLM app for misinformation, build a set of questions with known answers, check that every claim traces to retrieved sources, test what the model does when it does not know, and verify any code or package it suggests. OWASP LLM09 treats false but believable output as a security risk, not only a quality issue.
What does OWASP mean by misinformation?
OWASP's 2025 entry for LLM09 says models can produce false or misleading information that looks credible, and that this can lead to security breaches, reputational damage and legal liability. A main cause is hallucination, which it describes as the model generating content that seems accurate but is fabricated. Overreliance makes it worse, because people accept the output without checking.
The page lists four kinds of harm: factual inaccuracies, unsupported claims, misrepresented expertise, and unsafe code generation. It cites an airline chatbot that gave wrong information to travelers, after which the airline was successfully sued, and a case where fabricated legal cases caused problems in court. For engineers, the fourth kind matters most: the model suggests insecure or nonexistent libraries, and a developer installs them.
This post covers testing from the defender side. It contains no attack prompts.
How do I build a test set?
Start with your own product, not a public benchmark.
- Collect fifty to two hundred real questions your users ask. Include the ones where a wrong answer costs money or trust.
- Write the correct answer and the source for each, and have a second person confirm it.
- Add questions the system should refuse or answer with "I do not know", such as topics outside its documents.
- Keep a held-out slice that you never use to tune prompts, so a score gain means something.
Run the set on every model, prompt or retrieval change, and store the results next to the change. A drop of a few points after a model upgrade is the kind of regression you want to see before customers do.
How do I check grounding?
If your app retrieves documents, the claim is only as good as the match between the answer and the retrieved text. OWASP's prompt injection page names three checks it calls the RAG Triad: context relevance, groundedness, and question and answer relevance.
Test each separately. Did retrieval return the right passage? Does every statement in the answer appear in that passage? Does the answer address the question asked? Failures point to different fixes: retrieval, prompting, or the model.
How do I test code and package suggestions?
For any feature that writes code, run the output through the same checks as human code. Confirm that every suggested package exists in the registry you use, that the version is real, and that the name is the one the project intends. Run a dependency scan and a static scan on generated code before merge. Safeguard's free scan at gold.safeguard.sh/scan runs in the browser on up to 100 dependencies, which is a quick way to check a list a model gave you.
What mitigations does OWASP recommend?
The mitigation list includes grounding with retrieval from trusted sources, cross-verification and human oversight, automatic validation of key outputs, clear communication of risks and limits, secure coding practices, interface design that labels AI-generated content, and user training. Your tests should cover each control you claim to have. If you say a human reviews outputs, measure whether reviewers catch seeded errors.
What to do
- Write a golden question set with sources and a held-out slice.
- Add refusal cases for topics the app should not answer.
- Score groundedness against retrieved text, claim by claim.
- Verify every suggested package against the registry before install.
- Seed a few known errors into a review queue and measure the catch rate.
- Re-run the set on every model or prompt change and keep the history.
- Label generated content in the interface and state its limits.
Quick check
Is misinformation the same as hallucination?
Hallucination is a main cause. OWASP's misinformation risk is wider. It also includes biased or incomplete training data and the overreliance of people who trust the output.
Does retrieval augmented generation fix it?
It reduces it by grounding answers in trusted sources, and OWASP lists it as a mitigation. It does not remove the risk, so keep testing groundedness.
Sources
Last reviewed: 18 October 2026.
