Safeguard
Incident Analysis•

What does a major cloud outage look like, and how do you prepare?

Post-incident reports from AWS and Google Cloud show how big outages unfold: a small fault, shared dependencies, slow recovery. Plan for it calmly.

Safeguard Security Team
4 min read

A major cloud outage usually starts with a small internal fault, spreads through shared dependencies, and recovers in stages, not all at once. Two providers published detailed reports on recent events. They show the shape clearly and point to preparation steps you can take without predicting when the next one will come.

What did a real regional outage look like?

Amazon Web Services published a summary of a disruption in its Northern Virginia (us-east-1) region on 19 and 20 October 2025. AWS says the event ran from 11:48 PM PDT on 19 October to 2:20 PM PDT on 20 October, with three distinct periods of impact. First, DynamoDB had increased API error rates. Second, Network Load Balancer saw increased connection errors. Third, new EC2 instance launches failed for a period.

AWS traced the start to a latent race condition in DynamoDB's automated DNS management, which produced an incorrect empty DNS record for the regional endpoint that the automation failed to repair. Services that depend on DynamoDB, including some other AWS services, were affected as a result. AWS reported that it disabled the affected automation worldwide and listed fixes for the race condition, load balancer failover limits and EC2 recovery testing.

Notice the pattern: one record, one region, many dependent services, and recovery that happened in phases.

What did a global one look like?

Google Cloud's incident report for 12 June 2025 describes a different failure. The status page lists the incident as running from 10:51 to 18:18 US/Pacific, affecting many products across many locations. Google says a feature added to its Service Control policy checking system on 29 May did not have proper error handling and was not protected by a feature flag. On 12 June a policy change containing unintended blank fields replicated globally within seconds, and the binaries crashed in a loop. Google reports that recovery in some larger regions took longer because restarting tasks overloaded a dependent datastore, and that Service Control lacked randomized exponential backoff.

Different cause, same shape: a small change, wide blast radius through shared infrastructure, then a slow tail caused by everything restarting at once.

What does that mean for you?

You cannot prevent a provider outage. You can decide in advance what you do during one. The AWS Fault Isolation Boundaries whitepaper explains zonal, regional and global services and how to think about dependencies on each. The AWS Well-Architected Reliability Pillar covers designing, delivering and maintaining reliable workloads. Both are vendor documents, so apply the ideas to whichever provider you use.

Security teams should add one more concern: during an outage, your own tools may fail too. Scanning, secrets managers, identity, logging and ticketing often live on the same provider. If you cannot deploy a fix or rotate a credential, say so in your plan.

What to do

  1. Map your critical services to the provider regions, and global services, they rely on.
  2. List the dependencies of your security tooling: identity, secrets, logging, alerting, ticketing.
  3. Decide which functions need a second region or provider, and which can wait.
  4. Write a one page outage runbook: who decides, how you communicate, what you stop deploying.
  5. Keep contact details and runbooks outside the system they describe.
  6. Use retries with randomized exponential backoff in your own clients so recovery does not create a second wave.
  7. Rehearse once a year with a tabletop that removes one dependency at a time.
  8. After any outage, read the provider's report and record what you would change.

Quick check

Should every company be multi cloud?

No. It adds cost and complexity. Start by understanding single points of failure and decide where redundancy is worth it for your business.

Can I predict when a big outage will happen?

No, and this article does not try to. Plan for the type of failure, not the date.

Sources

Last reviewed: 18 October 2026.

Never miss an update

Weekly insights on software supply chain security, delivered to your inbox.

Self-healing security runs on Safeguard.

Your first fix PR is minutes away.

No sales call required, even your agent can complete the purchase over MCP.