Start with experiments that test what happens when a security control stops working, in staging first, with one owner and a stop condition. Five good starters: an expiring certificate, a revoked credential, a silent log pipeline, an unreachable secrets store, and a scanner that times out.
Ground rules before you start
The Principles of Chaos Engineering say to minimise blast radius and that containing fallout is the responsibility of whoever runs the experiment. For a first run that means staging, a working hours slot, a named person who can stop it, and a written hypothesis. Use stop conditions and guardrails and agree the abort signal in advance.
1. Let a certificate expire
Issue a short-lived certificate for an internal service and let it lapse. Questions to answer: do clients fail closed with a clear error, does monitoring alert before expiry, and can the on-call person renew it from the runbook without guessing? See the certificate expiry prevention checklist.
2. Revoke a credential mid-session
Create a test API key or session, use it, then revoke it. Measure how long the system keeps honouring it. Cached tokens and long-lived JWTs often keep working far longer than people assume. The result tells you your real revocation time.
3. Silence the log pipeline
Block the path between one service and your log collector. Does anyone notice? Does the service keep running, drop events, or buffer them? Is there an alert on missing logs, not just on bad logs? This one often finds that no one is watching for silence. A heartbeat check fixes that, see detecting silent failure of security tooling.
4. Make the secrets store unreachable
Cut access to your secrets manager for five minutes. Running services should keep working on what they already loaded. New deploys and restarts will fail, which is the part to examine. Does a restart fall back to a stale or default credential? That would be a failure open. The trade-offs are in when the secrets manager is down.
5. Time out the scanner
Make your dependency or code scanner hang in a test pipeline. Does the build block, pass silently, or pass with a warning? The right answer depends on the branch and the risk, and the team should have chosen it deliberately. See the CI gate decision.
What to record
For each run, write down the hypothesis, what happened, how long recovery took, and one fix with an owner and a date. An experiment with no follow-up work is theatre. If you can, schedule the same experiment again after the fix, which is what chaos engineering myths versus facts calls closing the loop.
Quick check
Should the first experiments run in production?
The principles favour production eventually, because that is where real conditions live. Build confidence in staging first, then widen carefully with a small blast radius.
How often should we run them?
Once a month is a realistic rhythm for a small team. Rotate through the five and automate the ones that prove stable.
Sources
Last reviewed: 19 October 2026.
