A kill switch you have never tested is a hope, not a control. Once a month, flip it in staging or on a low-risk feature, time how long it takes, check that the feature is really off and that nothing else broke, confirm the log entry, then turn it back on and time that too.
Why drills matter
Flags and switches rot. Permissions change, the person who knew where it lives leaves, a cache keeps serving the old value, a code path reads a different setting. The first time anyone finds out is during an incident, when time is short.
For design guidance, see the kill switch design checklist. This post covers the practice that keeps the design honest.
The drill, step by step
- Pick the target. Rotate through your switches. Use staging for the first run, then a low-risk production feature.
- Name the roles. One person flips, one watches metrics, one writes the timeline.
- Check access. Can the on-call engineer reach the switch at 3am from a phone or laptop with their normal login? If it needs a VPN or a special account, note it.
- Start the clock and flip. Record time from decision to effect.
- Verify the effect from outside. Call the feature as a user would. Do not trust the admin screen alone.
- Check propagation. Measure how long until every instance honours it. Caches and long-lived connections add delay.
- Check side effects. Did dependent features break? Did queued work get stuck? Did error rates move?
- Check the record. Is there an audit entry showing who, when and why?
- Restore. Turn the feature back on. Confirm it recovers cleanly and measure that time too.
- Write the findings. One action per problem, with an owner.
What good looks like
Set targets and write them down: for example, time to flip, time to full effect, time to restore. The numbers are yours to choose. What matters is that you can compare month to month and notice drift.
Common findings
- The switch works, but only for new sessions.
- Nobody on the current on-call rota has permission.
- The feature goes off, but a background job still writes data.
- The runbook link is dead.
- Turning it back on triggers a burst of queued requests that overloads a dependency.
That last one connects to load shedding and retry storms.
Who should run the drill
Rotate the flipper each month. If only one person has ever pulled the switch, you have a single point of failure in your process. Let the newest member of the on-call rota run it with a senior person watching, because that is the person most likely to be stuck at 3am. Keep the drill boring and the notes short, so it stays a habit rather than an event.
Choosing the right tool
Not every problem calls for a switch. See kill switch, feature flag or rollback to decide which to reach for.
Quick check
Can we drill in production?
Yes, on a low-risk feature with a small blast radius and an agreed stop condition. Start in staging.
How long should the drill take?
Often under an hour including the write up. If it takes much longer, that is itself a finding.
Last reviewed: 19 October 2026.
