Swapping the model behind a production agent is a release, not a settings tweak. The same prompts and tools can produce different actions with a new model. Pin the version, run a fixed test set including hostile inputs, canary the change on low risk work, and keep the old version ready to restore.
Why model changes are risky
Your code did not change, but behavior did. A new model may call tools more eagerly, interpret instructions differently, or resist injection better on some attacks and worse on others. NIST's agent hijacking work found that attack success depends strongly on the specific attacks and repeats used, so a model that looked safe on yesterday's tests needs fresh tests.
Step 1: pin the version
Use an explicit model identifier, not an alias that moves. Record it in config and in the agent's registry entry. If a provider retires a version, you want that to be a planned migration. See model deprecation plan.
Step 2: keep a fixed test set
Build a set that covers:
- Normal tasks. The ten to fifty things the agent does daily.
- Edge cases. Ambiguous requests, missing data, long inputs.
- Boundary tests. Requests that should be refused or escalated.
- Hostile inputs. Injection attempts in each place the agent reads, run several times each.
- Tool use checks. Does it call only permitted tools, with valid arguments?
Store expected results where you can, and compare tool calls, not just final text.
Step 3: compare, do not eyeball
Run old and new on the same set. Look at:
- task success
- unexpected tool calls
- refusals that changed
- cost and latency
- hostile input outcomes
Write down any regression and decide: fix, accept with a mitigation or block the change.
Step 4: canary
Route a small share of low risk traffic to the new model. Watch logs and approvals. Widen in steps. Keep high risk actions on the old version longest.
Step 5: roll back quickly
Make it a config change. Test that rollback works before you need it. Keep the old version available for a defined period.
Step 6: record it
Keep a change record: date, versions, test results, approver, and any accepted risk. This is also what an auditor or customer will ask for.
Who approves
For agents with sensitive access, the agent owner and a security reviewer should both sign. For low risk agents, the owner alone is fine.
Where Safeguard fits
Safeguard's policies, gates and guardrails stay in place when the model changes, which is a reason to enforce limits outside the model. Compliance and evidence automation can hold the change records.
Quick check
Is a minor version bump safe to skip testing?
No. Behavior can shift between versions, so rerun your fixed test set, including hostile inputs.
Why pin the model version?
So a provider side change does not alter your agent's behavior without a test and an approval.
Sources
Last reviewed: 19 October 2026.
