A failure mode review lists each thing your service depends on, asks what happens if it fails, and records whether the result is safe. In one hour with three or four people you can cover one service, find the unsafe defaults, and pick two or three experiments to run.
Before the meeting
Gather an architecture diagram or a rough sketch, the list of external calls the service makes, and the runbook if there is one. Invite someone who builds it, someone who runs it, and someone who thinks about abuse.
Step 1: list the dependencies (10 minutes)
Write every dependency on a board: databases, caches, queues, identity provider, secrets store, feature flag service, policy engine, logging and metrics, DNS, certificates, third party APIs, the CI scanner, the network path itself. Teams usually forget at least two. The forgotten ones are often region bound, see single region dependencies teams forget.
Step 2: ask the same four questions (25 minutes)
For each dependency, ask:
- What happens if it is down?
- What happens if it is slow?
- What happens if it returns wrong or stale data?
- What does the service do in each case: allow, deny, degrade or crash?
Write the answer in a table with three more columns: who notices, how, and how fast.
Step 3: mark the unsafe ones (10 minutes)
Circle any row where the outcome is a security downgrade: access granted without a check, logging dropped silently, validation skipped, a default credential used. These are the fail open cases, the weakness class MITRE catalogues as CWE-636. The decision method is in fail open or fail closed, and the terms are in fail-safe vs fail-secure.
Also circle rows nobody notices. Silent failures need heartbeats, see silent failure of security tooling.
Step 4: rank and choose (10 minutes)
Give each circled row a rough score for likelihood and for impact, say one to three each, and multiply. Do not argue over the numbers; the point is to get an order. Pick the top three and assign an owner to each.
Step 5: turn it into experiments (5 minutes)
For each top item, write a one line hypothesis and decide how to test it in staging. The method is in writing a chaos hypothesis.
What you leave with
A dependency list, a table of failure behaviours, three unsafe defaults with owners, and three experiments. Put the table in the repository next to the service so the next review starts from it.
Keep it short. A review that grows into a day long workshop will not get repeated, and repeating is where the value is.
Quick check
How often should we repeat it?
Whenever the architecture changes meaningfully, and at least once a year for services that matter.
Do we need a formal FMEA template?
No. A table with dependency, failure type, outcome, detection and owner is enough for most software teams.
Sources
Last reviewed: 19 October 2026.
