Retry storms: how retries turn a small fault into an outage

A retry storm happens when many clients retry a failing call at the same time and at several layers, so the struggling dependency receives far more traffic than before the fault. Fix it with capped attempts, exponential backoff with jitter, a retry budget, and retries at one layer only.

Why retries make things worse

A retry is a bet that the next attempt will succeed. When the failure is a brief blip, the bet pays off. When the failure is overload, every retry adds work to a system that is already behind.

AWS describes retries as "selfish" in its Builders Library article on timeouts and backoff, because each one spends more of the server's capacity to improve one caller's odds. During overload that can deepen the problem and delay recovery.

Layers multiply

The nasty case is stacking. Suppose a browser retries, the API gateway retries, the service retries, and the database driver retries. If each layer makes three attempts, one user action can become 81 calls to the database. The exact numbers depend on your stack, but the shape holds: attempts multiply, they do not add.

The Google SRE book gives the same advice. Avoid retrying at every layer, limit attempts per request, and do not retry permanent errors.

The four controls

1. Cap attempts. Two or three tries is usually plenty. Make the cap explicit in config, not a library default nobody read.
2. Back off. Wait longer after each failure, with a ceiling on the wait.
3. Add jitter. Randomise the wait so clients that failed together do not come back together. A cap alone can leave every client retrying at the same rate.
4. Budget retries. Allow retries only while a local token bucket has tokens, so retries stay a small fraction of normal traffic.

Pick one layer to own retries. Usually that is the layer closest to the caller that understands the operation, and every layer below it should fail fast.

The security angle

Retry storms are an availability problem, and availability is part of security. They also interact with controls you may rely on. A login service that retries a slow identity provider can lock out real users. A policy check that retries until it gets an answer can hold requests open and exhaust threads. Rate limits that count retries as new requests can start rejecting good traffic.

Idempotency matters too. Retrying a payment or a deletion without an idempotency key can do the action twice. Only retry operations that are safe to repeat, or make them safe first.

How to test it

Inject latency or errors into one dependency in staging and watch the call counts at the dependency, not just at the edge. If the dependency sees several times the normal request rate, you have stacked retries. A chaos experiment with a clear stop condition is a good fit here, see stop conditions and guardrails for chaos experiments.

For LLM calls the same rules apply, with extra cost at stake: retries, timeouts and circuit breakers for LLM API calls.

Quick check

Sources

• AWS Builders Library: Timeouts, retries and backoff with jitter
• Google SRE book: Addressing cascading failures

Last reviewed: 19 October 2026.