Rehearse first the scenarios where likelihood is real, impact is high and your recovery is least proven. Score each on those three, multiply or sort, and start at the top. For many software teams that means a leaked credential, a bad release, an expired certificate and a deleted repository before a full region loss.
Build the scenario list
Start with a list of what could go wrong, phrased as events:
- A cloud region becomes unavailable.
- A TLS certificate expires.
- A key or token is leaked.
- A repository is deleted or a branch overwritten.
- A maintainer account of a dependency is compromised.
- A bad dependency update reaches production.
- The identity provider is down.
- A database is corrupted or wiped.
- The CI system is unavailable.
- A key person is unreachable.
Add what your own history and your own architecture suggest.
Score each one
Use a three point scale, low, medium, high, for three questions:
- Likelihood. How often does something like this happen to teams like yours, or to you? Use your own incident history first. Do not invent statistics; if you lack data, a rough team estimate is fine, labeled as such.
- Impact. If it happened tomorrow, how bad: money, customers, safety, legal duties?
- Recovery confidence. Have you ever done the recovery? Is there a runbook, an owner and a tested path? Low confidence is a high priority.
Prioritize where recovery confidence is low and impact is high, even if likelihood is moderate. Rare and catastrophic events deserve at least a tabletop. Common but small events deserve automation, not drills.
Example outcome
The ranking is yours, but an illustrative result could look like this: expired certificates score high on likelihood and moderate on impact, with automation as the main fix. A leaked credential is moderate to high in likelihood and high in impact, so a rehearsal of the rotation order pays off. Full region loss is lower in likelihood and very high in impact with low confidence, so a tabletop and a documented decision rule come first, with a live failover drill later.
Choose the rehearsal type
- Read through. Someone not involved reads the runbook and lists gaps. Cheap and quick.
- Tabletop. A group walks through the scenario in conversation. CISA publishes free tabletop exercise packages with template objectives, scenarios, discussion questions, a slide deck, a feedback form and an after action report template. They are a ready starting point.
- Simulation in staging. Execute the steps against a test environment.
- Live drill. Run the real failover or restore under controlled conditions. Highest value, highest risk, so schedule carefully.
Match the effort to the score. Low scoring scenarios get a read through. The top few get a simulation or live drill.
Schedule and track
Plan one rehearsal a month. Keep a table with scenario, score, last rehearsed, owner and result. After each, file the follow up items as normal work, as described in turning postmortem actions into sprint work.
Revisit the ranking
Re-score twice a year, and after each real incident. A scenario you just recovered from may drop in priority, while a new dependency may add one.
Quick check
Should we always start with a full region loss?
Not necessarily. Start with a tabletop for it, and spend live drill time first on common events where your recovery is untested.
What if we have no incident history?
Use team estimates, label them as estimates, and revise after your first real events.
Sources
Last reviewed: 19 October 2026.
