Roll config and content changes out in rings: a small canary first, then wider groups, with bake-in time and an automatic stop between each. CrowdStrike's post-incident report names staged deployment as a finding for exactly this reason, because one bad content file reached a very large fleet.
What CrowdStrike changed
The company's root cause analysis states that each new Template Instance should be deployed in a staged rollout. It describes canary testing, then successive promotion to wider deployment rings, with additional bake-in time where telemetry is gathered before the next ring. It also says customers can now choose where and when Rapid Response Content updates are deployed.
Microsoft's account of the same day estimated that 8.5 million Windows devices were affected, which it described as under one percent of all Windows machines. Small share, large absolute number. That is the shape of risk that rings are meant to cut.
A ring plan you can copy
- Ring 0, internal: your own staff machines or a staging cluster. Push here first, always.
- Ring 1, canary: one to two percent of production, chosen to include a mix of regions and versions.
- Ring 2, early majority: around a quarter of production.
- Ring 3, everyone else.
Pick percentages that fit your fleet. The point is that each ring is bigger than the last and that something checks health before promotion.
What the gate between rings should check
- Crash and restart rates against the previous baseline.
- Error rates and latency on the paths the change touches.
- Whether the fleet is still reporting in. A silent fleet is a bad sign.
- A hard stop that needs no human to be awake.
Bake-in time matters. A change that fails only at the next scheduled event, such as the next notification from the operating system in the CrowdStrike case, needs a window long enough for that event to occur.
Make rollback real
Staging without rollback only slows the damage. Keep the previous version available, test the rollback path, and make sure it does not depend on the thing you just broke. If devices cannot boot, a cloud console will not help you. Plan an out-of-band recovery.
Ask vendors the same questions
If you run third-party agents that update themselves, ask: can we choose update timing? Is there a ring or canary option? Where are release notes? Many teams find they cannot answer these for their own critical software. Start with the agents that run with the highest privilege.
Quick check
Is a staged rollout too slow for urgent security content?
It can be tuned. Use a shorter bake-in for urgent content, but keep a canary ring and an automatic stop. Speed without a stop is what makes a bad push global.
How big should the canary be?
Big enough to hit varied hardware and configurations, small enough that losing it is survivable. One to two percent is a common starting point, not a rule.
Sources
- CrowdStrike, External Technical Root Cause Analysis, Channel File 291, 6 August 2024.
- Microsoft, Helping our customers through the CrowdStrike outage.
Last reviewed: 19 October 2026.
