A rollback plan for automated remediation answers five questions before anything breaks: what counts as a failure, who decides, how to undo the change, how to stop the bot, and how to learn from it. Write it once, test it once, and link it from your remediation policy.
1. Triggers: what counts as failure
Define specific triggers, not feelings. Examples to adapt:
- Error rate on the affected service rises above a threshold you set for a set period after deploy.
- A required health check fails.
- A customer facing feature using the changed library stops working.
- Build or deploy pipelines fail on the main branch after the merge.
- Two or more automated changes are reverted in a week.
Use your own service level objectives to set the numbers. Do not copy thresholds from elsewhere.
2. Decision owner
Name a role, not a person: the on call engineer for the service, with authority to revert without asking. Escalation goes to the service owner. Fast decisions matter more than perfect ones, since a revert is cheap.
3. Revert steps
For a squash merge, revert the single commit. For a merge commit, name the mainline parent.
git revert <commit>
git revert -m 1 <merge-commit>
Then run CI, verify the lockfile returned to the old versions with a frozen install, and deploy through the normal pipeline. The full walkthrough is in how to roll back a bad dependency upgrade merge.
Add items for changes that are not just code:
- Configuration or posture settings changed by the fix: record the previous values in the PR so they can be restored.
- Database migrations: state whether they are reversible.
- Container image tags: note the previous digest.
4. Kill switch
Write the exact steps to stop the automation:
- Pause the affected category, setting it back to manual.
- Pause schedules or integrations.
- Revoke or rotate the bot credential if compromise is suspected.
- Close open bot PRs, or label them hold.
Test these steps once on a non production repository. The blast radius guide lists related controls.
5. Post deploy watch
After merging an automated change, someone watches the relevant dashboards for an agreed period. Define who, where, how long and which signals. For low risk changes this can be short. For major bumps, extend it.
6. Communication
Decide who gets told when a rollback happens: service owner, security team, and anyone who relies on the fix. If the revert reintroduces a known vulnerability, say so and set an owner and date for the real fix. Do not leave an exposed version in place without a decision.
7. Review after the event
Within a week, hold a short blameless review. Capture cause, detection time, and one change to prevent a repeat: a new test, a tighter tier, a different strategy. Feed the lesson into the approval tiers.
One page template
Service and category:
Failure triggers:
Decision owner (role):
Revert method and who executes:
Non code items to restore:
Kill switch steps:
Watch period, dashboards, owner:
Who to notify:
Review date:
Quick check
How often should we test the rollback plan?
At least once when you set it up, and again when the tooling or ownership changes. A plan nobody has run is a guess.
Safeguard remediation categories ship set to Manual, so approval is the first control, and this plan is the last.
Last reviewed: 19 October 2026.
