On 19 and 20 October 2025, a latent race condition in the automation that manages DNS for Amazon DynamoDB left the regional endpoint in us-east-1 with no IP addresses. AWS says the event ran from 11:48 PM PDT to 2:20 PM PDT. It was a software fault, not an attack, and it spread to many other services.
What happened
AWS's post-event summary describes two parts in its DNS management system. A planner generates DNS plans. Three redundant enactors, one per Availability Zone, apply those plans to Route 53.
One enactor was delayed and applied an older plan after a newer enactor had already applied a newer one. A check made earlier did not catch that the plan had gone stale. The newer enactor then ran its clean-up of old plans and deleted the older plan, which removed every IP address for dynamodb.us-east-1.amazonaws.com. The system was left in a state that automation could not repair, so operators had to step in.
Why it lasted so long
DynamoDB itself recovered first. AWS lists its DNS information as fully restored at 2:25 AM and customer caches expiring by 2:40 AM. The rest of the day was recovery of the things that depend on it.
EC2 launches failed for hours because a subsystem that manages leases on physical hosts entered what AWS calls congestive collapse while re-establishing leases. Network Load Balancer health checks then failed in a loop, with nodes alternating between removed and restored. AWS says it disabled automatic health check failover for NLB at 9:36 AM, and normal EC2 operation returned at 1:50 PM. Some Redshift clusters stayed impaired until 4:05 AM on 21 October because their replacement workflows were blocked.
The pattern is common. The root fault is fixed in a few hours. The backlog, the retry storms and the dependent systems take the rest of the day.
What AWS says it is changing
AWS states that it disabled the DNS Planner and Enactor automation worldwide until the race condition is fixed and safeguards against bad plans are added. It also plans a velocity control that limits how much capacity a single NLB can remove during failover, a new test suite for the EC2 recovery workflow, and better throttling based on queue size.
Lessons you can apply
- Look for automation that can delete the last good copy of something. A clean-up job that runs after a newer actor finishes is a classic place for that.
- Rate limit your own recovery. Congestive collapse happens when everything tries to reconnect at once. Throttle work coming back in.
- Map hidden dependencies on one region. The AWS summary lists sign-in, support case creation, and container services among the affected parts. Know which of your own control paths run through a single region.
- Rehearse manual repair. The automation could not fix itself, so the first hour depended on people knowing the system.
- Write down how you tell an outage from an attack. This one was not an attack, but the first symptoms look similar.
Where Safeguard fits
Outages like this are an availability topic more than a vulnerability one. Safeguard's asset discovery and dependency scanning can help you see which services and libraries your applications depend on before an incident. You can try the free scan at gold.safeguard.sh/scan.
Quick check
Was the October 2025 AWS outage a cyberattack?
No. AWS attributed it to a latent race condition in DynamoDB's automated DNS management, per its own post-event summary.
Why did problems continue after DynamoDB recovered?
Dependent systems such as EC2 launches and Network Load Balancer health checks needed their own recovery, and AWS throttled work to stabilise them.
Sources
Last reviewed: 19 October 2026.
