A prompt regression test runs a fixed set of sample inputs against a changed prompt and compares the outputs with expectations before the change merges. It will not prove a prompt is safe, but it catches obvious breakage and gives reviewers evidence instead of opinion.
What it is for
Prompts fail quietly. A reworded instruction can drop a required output format, loosen a refusal, or change how the model treats pasted text. Nothing crashes. A regression set exists so these changes show up as a failing check rather than a customer complaint.
What belongs in the set
Aim for 20 to 50 cases to start. Group them.
| Group | Purpose | Example |
|---|---|---|
| Normal use | Prove the main job still works | A typical ticket to summarize |
| Format | Check structure the next system depends on | Output must be valid JSON with named fields |
| Edge input | Empty, very long, other languages | A blank field, a 20 page paste |
| Refusals | Things the app must decline | A request outside its scope |
| Hostile text | Instructions hidden in untrusted input | A document saying to ignore earlier rules |
Include every real incident you have had. A bug that happened once is a strong candidate for a permanent test.
How to grade outputs
Exact string matching rarely works because model output varies. Use checks that fit the case.
- Structure checks. Parse the output and verify fields exist. These are deterministic.
- Rule checks. The answer must not contain a forbidden phrase, a link to an unapproved domain, or a tool call that is not allowed.
- Reference checks. For narrow tasks, compare against a known good answer with a tolerance.
- Human review. For a small sample, a person reads the output. Do not skip this for high risk prompts.
Prefer deterministic checks wherever you can. If you use another model to grade, remember that it is also a component that can be fooled, and spot check its verdicts.
Run it as a gate
Add the suite as a pipeline step that runs when files in the prompts folder change. Fail the build on a structure or rule failure. Treat a human review sample as a required approval for high risk templates. Pin the model version in the test run so a provider update does not look like your regression.
Model output can differ between runs. If a case flips between pass and fail, run it several times and decide on a threshold, or tighten the prompt until the result is stable. Flaky tests train people to ignore the gate.
What it cannot prove
A passing suite means these inputs behaved. It says nothing about inputs you did not think of. Hostile text cases are samples of an open ended space, and the OWASP data and model poisoning guidance notes that backdoors can leave behavior untouched until a trigger appears, which testing by sample may miss. Treat the suite as a floor and pair it with limits in code: narrow tool permissions, output validation and human approval for risky actions.
Keep it healthy
- Add a case whenever something breaks.
- Review the set quarterly and drop cases that no longer reflect the product.
- Store expected results in the repository next to the prompts.
- Record who approved a change to expectations, since loosening a test is also a risk.
For the surrounding process, see how to version and review prompt templates in Git.
Quick check
How many test cases are enough?
There is no fixed number. Start with a couple of dozen across the groups above and grow the set from real failures.
Should the gate block merges on a failed hostile text case?
For prompts that control tools or data access, yes. For low risk prompts, a warning with required reviewer acknowledgement can be enough.
Sources
Last reviewed: 19 October 2026.
