Keep a frozen test set and rerun it after every announced or suspected model change. Compare flags, misses and wrong flags against the last accepted run. If results got worse on cases you care about, escalate to the vendor and, if you can, pin the previous version until it is fixed.
Why this needs its own process
With a rules-based scanner, an update changes rules you can read in release notes. With an AI reviewer, an update can swap the underlying model, change instructions the vendor wrote around it, or alter how context is chosen. Behaviour may shift in ways no release note lists. Sometimes the tool improves overall and gets worse on exactly the class you rely on.
NIST's AI Risk Management Framework organises its work into Govern, Map, Measure and Manage functions, and its generative AI profile, NIST AI 600-1, helps organisations identify risks specific to generative AI. Measuring on a schedule and managing change fits that approach, even if you never mention the framework by name.
What to freeze
- A labelled test set of vulnerable cases, safe twins and plain changes.
- The scoring script or spreadsheet, so scoring does not drift.
- Your tool configuration, exported or recorded.
- The results of the last accepted run, with date and version if shown.
Do not edit cases to make a new run pass. If a case is wrong, fix it, record the change, and rescore the old run too.
Triggers to rerun
| Trigger | Notes |
|---|---|
| Vendor announces a new model or major version | Rerun before you accept it where you can choose |
| Vendor changes data or hosting arrangements | Also a data handling review |
| Your configuration changes | Thresholds, rule sets, scope |
| Developers report a change in behaviour | Treat as a signal even without a notice |
| Calendar | Monthly or quarterly if nothing else fires |
Silent changes are the reason for the calendar trigger.
Compare the runs
Build one table with each case as a row and columns for last run and new run. Highlight three kinds of difference:
- A previously caught vulnerable case now missed. Most serious.
- A previously quiet safe case now flagged. Adds noise.
- A changed severity or a materially different explanation.
Then read the changes, not just the totals. One lost catch in authorization code may matter more than five fewer wrong flags elsewhere. Because outputs vary run to run, use a few repeats on any case that changed, as in the repeat test, before concluding anything.
What to do about a regression
- Report it to the vendor with the specific cases and both outputs.
- Ask whether you can pin the earlier model or configuration.
- Tighten other controls for the affected class, such as a deterministic check or required human review.
- Record the issue and the decision in your vendor file.
Contract terms that help
Ask for advance notice of model changes, a changelog that names the model or its provider, the ability to pin or defer versions for a period, and notice when sub-processors change. Not every vendor will agree. Knowing which terms you did not get tells you how often to test.
Keep the history
Store each run with its date. Over a year, the series shows whether quality drifts and gives you real evidence for renewal and audits, as described in evidence auditors want.
Quick check
How often should I rerun the test set?
After any change, and on a calendar such as monthly or quarterly to catch silent changes.
What if the vendor will not tell me when the model changes?
Test more often and keep deterministic controls as the hard gates. Note it in your vendor risk record.
Safeguard's policies, gates and guardrails let you keep hard checks that do not change with a model update. You can scan a repository free at gold.safeguard.sh/scan.
Sources
Last reviewed: 19 October 2026.
