Before writing a security review prompt, write a one page rubric: what must be checked, what counts as a finding, how severity is set, and what a correct answer looks like on a few past cases. The rubric tells you what to put in the prompt and gives you a way to tell whether a change made it better.
Why the rubric comes first
Prompt tweaking without a target is guesswork. Each edit feels like improvement until a different input breaks it. A rubric turns "seems better" into a comparison.
Anthropic's guidance has a related test: show the prompt to a colleague with minimal context and see if they would be confused. A rubric forces you to put the context into words, and what you can write for a colleague, you can write for a model.
What goes in the rubric
Checks. The specific questions for this kind of review. For a web API: is every endpoint authorized for the specific record, is input validated at the boundary, are queries parameterized, are secrets absent.
Finding definition. What qualifies, and what does not. For example, test code and generated files are out of scope, and a theoretical issue with no reachable path is a note, not a finding.
Severity rules. Simple and written down. Which situations are urgent, which are scheduled.
Evidence requirement. Every finding names the file, line, and a quote.
Answer key. Five to ten past changes with the correct outcome, including clean ones. This is your test set.
Turning it into a prompt and a test
Put the checks, the definition, and the evidence rule into the prompt. Keep the answer key out of it, and run the prompt on those cases afterwards. Score three things: findings the model missed, findings it invented, and findings it got right with the wrong severity.
Record the scores with the prompt version. When you change the prompt, rerun the set. If a change fixes one miss and creates two inventions, you will see it.
Keep the set honest
- Include cases from your real code, not only textbook examples.
- Include clean changes. A set of only bad code rewards a model that flags everything.
- Refresh it when you find a real miss in production, so that miss becomes a test.
- Remove sensitive details before reuse. See what not to paste into an LLM.
Small is fine
Ten cases take an afternoon to build and pay back quickly. For a larger program, see building an eval suite for security LLM workflows.
Use the rubric for humans too
The same document helps new reviewers and gives an audit trail of what your review covers. If the model and a person disagree, the rubric is the tiebreaker, and a gap in the rubric is a reason to update it.
Quick check
How many answer key cases do I need?
Start with about ten, mixing real issues, clean changes, and borderline cases. Add to it over time.
Sources
Last reviewed: 19 October 2026.
