Google's Workbook describes an incident where an empty list was treated as no filter, so a retried decommission job wiped drives across all of its satellite racks. The lessons apply to any automation, including security remediation: make commands safe to repeat, rate limit them, cross-check requests and keep blast radius small.
What the case study says
The Workbook describes a routine rack decommission. A bug in maintenance automation, combined with insufficient rate limits, caused thousands of servers carrying production traffic to go offline at once. After the first run partly failed, the decommission was retried. This time a lookup returned an empty list, and the API treated an empty list as no filter, not as "act on no machines". Within minutes, disks of satellite machines globally were erased. Users saw a slight latency increase, and the team spent about two days reinstalling.
The write-up lists the process failures: the admin server lacked sanity checks on commands, the downstream system did not object to a missing constraint, the decommission flow did not cross-check requests against other data, and it was not rate limited. It also says all commands should be idempotent, or at least fail safe on repeat invocation.
Lessons as design rules
| Failure | Design rule | Security automation example |
|---|---|---|
| Empty input meant "everything" | Treat empty as nothing, and require an explicit scope | A cleanup job that revokes "all tokens" when the filter returns none |
| Retry repeated a destructive step | Make steps idempotent or fail safe on repeat | A rotation script that runs twice |
| No cross-check | Compare the request with a second source | Compare a mass block list with planned changes |
| No rate limit | Cap speed and count per run | Limit how many resources a job can change |
| Fast, wide blast radius | Roll out in stages | Apply to one group first |
The case also shows what went right: capacity planning let traffic shift to core clusters, and automatic mitigation drained failing satellites. Those are mitigation controls, and they limited customer impact.
Why it matters for remediation automation
Automation is attractive because it acts faster than people. That is also its risk when the input is wrong. For any automated fix, ask the questions the case raises:
- What happens if the input list is empty or malformed?
- What happens if the job runs twice?
- Is there a second check that would object to an unusual request?
- How many changes can one run make before it stops and asks?
- Can a human see and approve a change before it spreads?
Steps to review your own automation
- Inventory jobs that change things at scale: rotations, cleanups, mass updates, automatic fixes.
- For each, test with empty, duplicate and oversized inputs in a safe environment.
- Add a maximum change count and a dry run mode.
- Add a cross-check against an independent source.
- Write down what the rollback is. See feature flag kill switch for security incidents.
- Run a replay of an automation incident so responders practise it.
Try it yourself
Click through it at your own pace, or open it full screen. Look at how a proposed fix is shown for review before anything changes.
How Safeguard fits
Safeguard's autonomous remediation opens fix pull requests for review. It is off by default, with every category set to Manual, so changes are reviewed by people. A free scan is available at https://gold.safeguard.sh/scan.
Quick check
What does idempotent mean?
An operation is idempotent if running it twice has the same result as running it once.
Should security automation require approval?
For wide or destructive changes, yes, at least until the job has proven safe in staged rollouts.
Sources
Last reviewed: 19 October 2026.
