In the first 30 minutes of a region outage, confirm the problem is the provider and not your own change, name one incident lead, decide whether failing over costs less than waiting, and tell customers what you know. Do not start rebuilding anything until you know whether data is at risk.
Minutes 0 to 10: confirm and assign
Check the provider status page and your own recent deploys at the same time. A bad release and a regional fault look alike in dashboards. If you shipped something in the last hour, treat your own change as a suspect until it is cleared. The companion post outage or attack: how to tell in the first hour covers that split in more detail.
Name one incident lead and one person who writes down what happens, with timestamps. The lead decides. Everyone else reports. Without this, five engineers try five fixes and nobody knows which one worked.
Open a channel for the incident and keep it free of side chatter. Pin the current status in one line and update it when it changes.
Minutes 10 to 20: scope and the failover question
List what is down, what is degraded, and what still works. Be specific: login, checkout, API, background jobs, data pipelines. A short written table beats a mental picture.
Then ask the failover question. AWS notes in its disaster recovery guidance that automatically initiated failover should be used with caution, because recovery time and recovery point are never zero, and a false alarm still costs you availability and data. Many teams therefore keep failover manual but automate the steps, so that starting it is like pushing a button.
The same guidance makes a point that matters during a regional event: control planes, the APIs used to configure the environment, usually have lower availability goals than data planes, which serve live traffic. A runbook that depends on creating new resources in the failing region, or on a console that is itself degraded, may not work when you need it. Write your failover steps so they rely on things that are already running.
Failing over is worth it when the expected outage is long, the standby is ready, and data replication lag is known. Waiting is better when the provider reports a fix is close and your standby has not been tested lately. If it has never been exercised, say so in the incident notes. That is a finding, not a failure.
Minutes 20 to 30: protect data and communicate
Before you switch traffic, find out how far behind your replicated data is. Promoting a replica with unknown lag can lose recent writes. Record the lag and the time you promoted it, because you will need both when you reconcile later.
Pause jobs that write to systems you cannot see. Retry storms from queues and cron jobs can make recovery slower and corrupt state.
Post a customer update now, even if it is short. A useful update has four parts: what customers see, what you know, what you do not know yet, and when the next update will come. Avoid naming a cause until you have one. Guesses become promises.
What to prepare before the day
- A written failover decision rule with a named owner.
- A list of dependencies that live in one region only, such as the CI system, the secrets store, and the status page itself.
- Replication lag shown on a dashboard, not discovered during the incident.
- A tested path to deploy to the standby region without the primary one.
- Service quotas in the standby region high enough to scale up to production load.
Quick check
Should we fail over automatically when a region degrades?
Often not. Automatic failover on a false alarm costs availability and data, so many teams automate the steps and keep the trigger manual.
What is the first thing to confirm?
That the fault is the provider's and not your own recent change. Check the provider status page and your deploy history side by side.
Sources
Last reviewed: 19 October 2026.
