On 8 October 2026, Google Cloud Storage customers in us-central1 (Iowa) saw elevated latencies and errors on upload and write operations. The problem returned on 9 October in the us-central1-b zone. Google published a preliminary report on 9 October and a final incident report on 14 October.
Everything below comes from Google's own incident page unless noted. Google's status page shows times in US Pacific, which is UTC minus 7 on these dates, and the conversions are ours.
What happened
Google's incident header lists the event as running from 8 October 04:00 Pacific to 9 October 09:46 Pacific. The report itself breaks that into two parts.
- First event: 8 October, 04:00 to 16:30 Pacific (11:00 to 23:30 UTC), 12 hours 30 minutes.
- Recurrence: 9 October, 04:05 to 09:46 Pacific, in us-central1-b, 5 hours 41 minutes.
- Total: 18 hours 11 minutes, per the report.
The start time is not consistent inside Google's own material. The preliminary report says 04:00, the live status updates say the issue began at 05:19, and the final report says it was detected at 04:50. The first public status update was posted at 11:24 Pacific on 8 October, so customers saw an acknowledgement several hours after the reported start.
What customers saw: Google says they "primarily observed HTTP 503 UNAVAILABLE errors on WriteObject operations", with read operations minorly affected. Google states other Google Cloud services were unaffected. No workaround was listed.
The root cause, per Google
The final report names three separate things.
- A configuration update to the load balancing layer for a "traffic sorting optimization" inadvertently bypassed the standard capacity-aware routing logic. Write traffic piled onto a small set of low capacity nodes, which ran out of resources and timed out.
- A separate issue in the underlying database service caused metadata processing errors for part of the window.
- The 9 October recurrence came from an incomplete rollback of an upload planner feature, which broke load balancing again and overloaded a few smaller nodes. It ended when the load balancing configuration change was fully reverted.
Google also says attempts to add emergency regional capacity were blocked by regional capacity limits. Engineers drained traffic from overloaded nodes, disabled the optimization and rolled back a recent metadata database update.
What is confirmed, reported and unknown
- Confirmed by Google: the dates, the error type, the three causes above, the preventative actions below.
- Reported by third parties: IncidentHub lists Torq, Temporal, Elastic Cloud and Mixpanel as downstream services that recorded impact. We did not check each vendor's own status page, so treat that list as unverified.
- Unknown: how many customers were affected, how much data was written late or lost, and whether anyone received service credits. Google's report gives no counts.
Why it matters
This was not an attack. A routine performance change, tested and shipped by a very large operator, removed a safeguard the system depended on. The fix was then rolled back only partly, and the fault came back the next day.
For customers the lesson is about dependency, not blame. Object storage sits under uploads, exports, backups, logs and build artifacts. When writes fail, the first things to break are often the ones nobody listed as critical.
What to do now
- Find every service that writes to GCS in us-central1. Search your Terraform, bucket lists and application config for the region.
- Check what each writer does on a 503. Does it retry with exponential backoff and jitter, queue the write locally, or drop it?
- Look at your own logs for 8 and 9 October. Count failed writes and confirm none were silently discarded.
- Decide which workloads can accept a write delay and which need a second location. Dual region or multi region buckets are a design choice you make per bucket, and they cost more.
- Add the provider status feed to the same place your on call team already watches, and note that the first update arrived hours after the reported start.
How this could have been avoided
Google cannot hand its customers a fix for its own change process, but its report points at the controls that would have narrowed this.
- Roll out in small regions first. Google lists this as a preventative action: validate load balancing changes in smaller regions before large, highly utilized ones.
- Make capacity limits non bypassable. The change skipped capacity-aware routing. Google says it will enforce capacity constraints so traffic cannot bypass them.
- Complete rollbacks and verify them. The second outage came from a partial rollback. A rollback checklist that confirms the system is back to the prior known state would have caught it.
- Customer side: do not make one region a single point of failure. A write path with a queue and a second bucket location turns an 18 hour outage into a delay.
Try it yourself
Pick one write heavy service and write down, in one sentence, what happens to a user action when the storage write returns 503. If you cannot answer, that is your first test to run.
Click through it at your own pace, or open it full screen.
Quick check
Why does the report say the second outage happened on 9 October?
Google says an upload planner feature was only partly rolled back. That broke load balancing again and overloaded a few smaller nodes in us-central1-b until the load balancing change was fully reverted.
Was this a security incident?
Google's report describes a configuration change and a database issue, not an attack. It is still a resilience event for anyone who depends on GCS writes.
