On March 21, 2025, Cloudflare's R2 object storage went down for 1 hour and 7 minutes, from 21:38 to 22:45 UTC. The cause wasn't an attacker — it was a routine credential rotation. Engineers ran wrangler secret put and wrangler deploy without the --env production flag, which pushed new R2 Gateway credentials to a development Worker instead of the production one. When the old credentials were then deleted as part of the rotation, production lost its authentication to storage entirely: 100% of write operations failed, and roughly 35% of reads failed globally. Cloudflare published a full post-incident writeup of exactly what happened. It's a textbook case for a discipline most teams still treat as optional: security chaos engineering, the practice of deliberately breaking your own auth, secrets, and network controls in a controlled way to find out whether they actually fail safely — before a routine Tuesday-afternoon rotation does it for you in production. This post covers where the discipline came from, what a real security chaos experiment looks like, and which controls are worth attacking first.
Where does security chaos engineering come from?
Security chaos engineering descends directly from Netflix's Chaos Monkey, open-sourced in 2012 to randomly terminate production instances and force engineers to build for failure. Netflix extended the idea in 2017 with ChAP (Chaos Automation Platform), described in a Netflix TechBlog post and a companion paper on arXiv (1702.05849): ChAP routes a small, statistically controlled slice of live production traffic into paired experiment and control clusters, injects RPC-level failures into the experiment cluster, and ties the whole thing to an automatic circuit breaker keyed to an error budget, so a bad experiment halts itself before it becomes a customer-facing incident. The security-specific application of this idea was formalized in the 2023 O'Reilly book "Security Chaos Engineering: Sustaining Resilience in Software and Systems," by Kelly Shortridge and Aaron Rinehart. Its core argument: security controls are usually verified by point-in-time audits and paper reviews, not by actually triggering them under realistic failure conditions — which means the first real test of whether your auth fallback or secrets rotation works correctly is often the day it's needed for real.
What's the difference between chaos engineering and a penetration test?
A penetration test asks whether an attacker can get in; a security chaos experiment asks whether your own controls behave correctly when something inside your system fails or changes state. A pen test is adversarial and episodic — usually an outside team, run once or twice a year, looking for a way through the perimeter. A chaos experiment is internal, continuous, and non-adversarial by design: you already know exactly what you're breaking (a credential, a firewall rule, an IAM policy) because you're the one breaking it, and the question isn't "can someone get in" but "does the system degrade the way the runbook says it will." The Cloudflare R2 incident is a clean illustration of the gap between the two: no attacker was involved, no perimeter was breached, and a pen test would never have surfaced the failure. The bug was entirely about how a credential rotation interacted with environment targeting — precisely the class of failure a chaos experiment run against a staging deploy, with the same wrangler flags and the same rotation sequence, would have caught before it touched production.
Why is secrets rotation one of the highest-value experiments to run?
Secrets rotation is one of the highest-value chaos targets because rotation is rare enough that teams rarely rehearse it, yet catastrophic enough that a mistake removes access instead of narrowing it. Most credentials get set once and left alone for months or years; when rotation finally happens, it's often the first time the exact sequence — issue new credential, deploy it to every consumer, confirm receipt, then revoke the old one — has been exercised outside a runbook document. The Cloudflare incident shows what breaks when that sequence has an ordering bug: the new credential landed on the wrong target, and the old credential was deleted regardless, so there was no working credential anywhere in production for over an hour. A rotation chaos experiment inverts the risk: intentionally revoke a non-production credential, or deploy a rotation to a scoped canary environment using the exact tooling and flags used in production, and watch whether monitoring catches the gap, whether the rollback path actually restores the old credential, and how long detection takes. Running this quarterly against low-blast-radius environments is dramatically cheaper than discovering the gap during a real rotation.
Which other security controls are worth deliberately breaking?
Beyond secrets rotation, three other control classes reward the same treatment: authentication fallback paths, network segmentation policies, and rate limiting or circuit breakers. Authentication chaos means forcing your identity provider or session store to time out or return errors on a canary path and confirming the system fails closed (denies access) rather than failing open (grants it) — a distinction that has caused real outages and real breaches industry-wide when misconfigured. Network policy chaos means deliberately applying an overly permissive or overly restrictive network policy in a scoped namespace and confirming your admission and monitoring layers actually flag the drift, rather than assuming a Kubernetes NetworkPolicy or security group rule is enforced just because it's committed to a repo. Rate-limit and circuit-breaker chaos, the closest descendant of Netflix's original ChAP work, means injecting synthetic load or forced errors against an auth or token-issuance service to confirm the circuit breaker trips before the service falls over entirely, taking every downstream consumer's auth checks down with it.
How do you run a security chaos experiment without causing the incident you're trying to prevent?
You run it safely the same way ChAP does: small blast radius, an automatic abort condition, and a control group to compare against. Concretely, that means picking a single non-critical service or a canary slice of traffic, defining in advance the exact signal that ends the experiment immediately (an error-rate threshold, a specific alert firing, a customer-facing symptom), and never running the first iteration of any experiment directly against production. Shortridge and Rinehart's book frames this as building a hypothesis first — "if we revoke this token, monitoring should alert within 60 seconds and the fallback path should deny access" — and then testing whether reality matches the hypothesis, rather than breaking things freeform and hoping something interesting happens. Every experiment should also have a named owner, a rollback script tested independently of the experiment itself, and a written record of the outcome, because the value compounds only if failures found in staging get fixed before the next rotation cycle, not just documented and forgotten.
How does Safeguard fit into a security chaos program?
Safeguard doesn't run chaos experiments — it isn't a fault-injection or resilience-testing platform, and this piece shouldn't be read to claim otherwise. What Safeguard does is complementary at the boundaries of a chaos program. Its secrets scanning verifies every finding against the issuing service before flagging it — an AWS key against sts:GetCallerIdentity, a Stripe key against /v1/balance — and its remediation playbook can revoke and rotate a live credential through your secret manager, which is the same rotation mechanism a chaos experiment would want to exercise deliberately rather than encounter live for the first time. Its policy guardrails, enforced as YAML rules at CI, registry, and admission, are the automated check that should catch a misconfigured deploy target or an unsigned image before it reaches production — the class of control a network-policy or deploy-target chaos experiment is specifically designed to stress-test. Neither capability injects failure on your behalf; both are the instrumentation you'd want in place to detect and recover from the failures your own chaos experiments are designed to surface.