DNS turns names into addresses. BGP tells networks how to reach those addresses. When either fails, everything built on top looks broken at once. Three companies published detailed accounts of such failures, and this post summarises what they say.
Facebook, 4 October 2021
Facebook's engineering post of 5 October 2021 says a command meant to assess backbone capacity unintentionally disconnected all backbone connections. A bug in the audit tool prevented it from stopping the command. DNS servers withdrew their BGP advertisements when they could not reach the data centers, so, in the post's words, "our DNS servers became unreachable even though they were still operational."
Recovery was slowed because normal and out-of-band access were both down, so engineers had to travel to data centers, which are hard to get into by design. The post also says loss of DNS broke internal tools used to diagnose the problem. Services were restored gradually to avoid a traffic surge.
Cloudflare 1.1.1.1, 27 June 2024
Cloudflare reports that at 18:51 UTC an unrelated network began announcing 1.1.1.1/32, a more specific prefix than Cloudflare's 1.1.1.0/24, which pulled traffic to it. Cloudflare's ROA authorised only its own AS up to /24, so the announcement was invalid, and Cloudflare's own routers rejected it. Many other networks accepted it. A separate route leak of 1.1.1.0/24 continued until 02:28 UTC on 28 June, about 7.5 hours after the start.
Cloudflare states that "the impact of a BGP hijack is greatly limited" when origin validation is enforced. Origin validation does not catch a leak, because the origin is valid, and path validation (ASPA) was still a draft at the time.
AWS us-east-1, 19 and 20 October 2025
AWS's summary says DNS failures for DynamoDB began at 11:48 PM PDT on 19 October. Cause: a "latent race condition in the DynamoDB DNS management system" left the regional endpoint with an empty DNS record, and operators had to intervene. DynamoDB recovered by 2:40 AM PDT, but knock-on effects on EC2 launches and load balancers continued until 2:20 PM PDT on 20 October, about 14.5 hours in all. Existing EC2 instances were not affected. AWS disabled the DNS automation worldwide pending a fix.
What the three share
Each failure was triggered by a change or a latent bug, not an attacker. Each was prolonged by dependence: tools that needed DNS, access that needed the network, services that needed a recovering service.
What to do now
- Use more than one DNS provider for your public zones, and keep TTLs short enough to allow a switch.
- Keep out-of-band access to critical systems that does not depend on your own DNS or network.
- If you run your own network, enforce RPKI origin validation and publish ROAs for your prefixes. RFC 6811 defines Valid, Invalid and NotFound states.
- Hard code a few critical addresses in runbooks so you can work when name resolution fails.
- Review services that resolve a single regional endpoint, and test what happens when it returns nothing.
- Roll out changes in stages with a fast rollback.
How this could have been avoided
For Facebook, the audit tool should have blocked the command and out-of-band access should not have depended on the backbone. For the Cloudflare event, wider route origin validation and prefix length filtering would have limited the hijack. For AWS, protection against applying outdated DNS plans is the fix AWS itself names.
Why it matters
You cannot prevent a provider's routing or DNS fault. You can make sure your own diagnosis and recovery do not rely on the same path.
Quick check
Why were Facebook's DNS servers unreachable though still running?
They withdrew their BGP advertisements when they lost contact with the data centers, so the internet could no longer route to them.
Does route origin validation stop a route leak?
No. The origin is valid in a leak. Cloudflare says path validation such as ASPA is needed.
