Before acting on an LLM's security finding, confirm that the quoted code exists, that untrusted input can reach it, that the problem reproduces, that the proposed fix works and breaks nothing else, and that you record the decision. A finding is a lead until a human or a test turns it into a fact.
Why a checklist
Language models write fluent explanations whether or not the underlying claim is right. Anthropic's own guidance says that techniques to reduce hallucinations do not eliminate them, and that critical information should still be validated. A repeatable checklist keeps the validation from depending on how convincing the explanation sounds.
The checks
1. Does the quoted code exist? Search the repository for the exact text. If the model quoted code that is not there, discard the finding. This is the cheapest and most useful filter.
2. Is it in scope? Production code, a test fixture, a generated file, or a vendored copy. Many invented or irrelevant findings land in code that never ships.
3. Can untrusted input reach it? Trace backward from the line to a source such as a request, a file, a queue message, or a header. If you cannot find a path, the finding may still be real in a future change, but it is not urgent.
4. Does a control already cover it? Look for validation, encoding, a framework default, or an authorization layer upstream. Models often review a function in isolation and miss these.
5. Can you reproduce it? For serious findings, write a failing test or a request that demonstrates the problem in a safe environment. A reproduction ends the debate in either direction.
6. Does the proposed fix work? Apply it, run the test from step 5, and run the existing suite. Read the diff for unrelated edits. AI suggested fixes sometimes remove a check instead of correcting it. See the AI-written pull request review checklist.
7. Record the decision. Note whether the finding was confirmed, rejected, or deferred, and why. That record feeds your measurement of how well the model performs on your code.
Handling disagreement between runs
If the same prompt reports the issue in one run and not another, go to step 3 by hand. Inconsistency is a signal about confidence, not a vote. Anthropic describes comparing multiple runs as a way to spot hallucinations.
Keep numbers
Over a month, track confirmed, rejected, and unclear. If most rejected findings fail at step 1, fix the prompt with a quote rule. If most fail at step 4, give the model more context about your controls. The failure stage tells you what to change.
Quick check
Do I need all seven steps for every finding?
No. Steps 1 to 3 are quick and filter most noise. Use steps 5 and 6 for findings that would change a release or a priority.
Who should sign off a confirmed finding?
The owner of the code, with a security reviewer for anything touching authentication, data access, or money.
Sources
Last reviewed: 19 October 2026.
