Recovering services start cold. Caches are empty, connections are not set up, and waiting clients all return at once. That surge can overload the service again. Bring systems back in stages, ramp traffic slowly, and test the restart path in advance rather than discovering it during recovery.
What cold start does
Google's Site Reliability Engineering book describes the effect. Processes are often slower to respond right after starting than in steady state, because of required initialization, connections set up on first use, and in some languages runtime optimizations that arrive later. Caches matter too. In a service where most requests are answered from cache, an empty cache turns almost every request into an expensive one.
The book lists situations that lead to a cold cache: turning up a new cluster, returning a cluster to service after maintenance, and restarts. It draws a distinction between a latency cache, where the service can handle expected load with an empty cache, and a capacity cache, where it cannot. Caches added for speed can quietly become hard dependencies.
Its suggested responses include overprovisioning, rejecting requests when overloaded, testing how the service behaves after a large restart, and increasing load slowly on a cluster, since the small initial request rate warms the cache before more traffic is added.
A real example of staged return
Meta's report on its October 2021 outage notes a related concern. After the backbone was restored, the team knew that switching services back on all at once could cause a new round of crashes from a surge in traffic. It described dips in power use in the tens of megawatts at individual data centers and the risk of suddenly reversing them. It said the company's long running "storm" drills, which simulate a major failure by taking a service, data center or region offline, had prepared it to bring services back while carefully managing the rising load. Services came back without further systemwide failures.
Why the surge happens
| Cause | What it looks like |
|---|---|
| Empty caches | Every request reaches the database |
| Clients retrying at once | A wave of requests at the moment of recovery |
| Connection setup | Pools filling, handshakes piling up |
| Autoscaling lag | Capacity trails the demand |
| Downstream dependencies restarting too | Slow answers feed more retries |
A ramp up plan
- Write the restart order. Dependencies first, then the services that rely on them. Include who confirms each step.
- Start with a fraction of traffic. Use your load balancer or routing to send a small share, watch error rates and latency, then increase in steps.
- Warm caches deliberately. Replay a sample of typical requests or preload key entries before opening the door.
- Ask clients to back off. Retries with exponential backoff and random jitter spread returning load over time. Check your own clients and any you control.
- Keep load shedding ready. A service that rejects excess requests quickly is better than one that slows for everyone. Our post on load shedding versus rate limiting explains the difference.
- Hold the incident open until steady state. Our post on verifying recovery before declaring the incident over lists the checks.
Rehearse it
The SRE book says that load testing a component until it breaks and testing its overload failure mode is perhaps the most important step in avoiding cascading failures. Add a restart test to the schedule: stop a non-critical environment, start it, and measure how long it takes to reach normal latency. Record the number and use it when you set recovery targets, as described in our post on what a restore really takes.
A worked example: restarting a service with a cold cache
This example is hypothetical. A service normally handles 2,000 requests per second, with a cache absorbing 95 percent of reads. A maintenance event empties the cache. When the service returns, the database receives roughly twenty times its usual read traffic, since 95 percent of reads now miss instead of 5 percent. Under that load the database slows, requests time out, clients retry, and the service falls over again.
This is simple arithmetic on the stated numbers: the share of reads that reach the database goes from 5 percent to 100 percent. The example shows why the Google SRE book separates a latency cache, which merely improves speed, from a capacity cache, which the service cannot do without.
| Approach | What happens | Outcome |
|---|---|---|
| Open to all traffic at once | Database overloaded | Second outage |
| Ramp traffic in steps of 10, 25, 50, 100 percent | Cache warms at each step | Stable return |
| Preload hot keys, then ramp | Faster warm up | Shorter recovery |
| Shed excess requests while warming | Some users see errors, service survives | Controlled degradation |
Step by step: a restart checklist
- Know your dependency order. Draw it: storage and databases first, then caches and queues, then services, then the edge.
- Start components in that order and wait for each to report healthy by a local check, as described in our post on shallow versus deep health checks.
- Before sending traffic, preload the cache with the most common keys, or replay a sample of recent requests against the service.
- Route a small share of traffic. Watch latency, error rate, database load and cache hit rate.
- If the numbers stay healthy for a few minutes, raise the share. If not, hold or reduce.
- Tell clients to back off if they have been retrying. Check that your own clients use exponential backoff with random jitter, so returning load spreads out.
- Keep load shedding on until the service reaches normal behavior.
- Record the time of each step. The record becomes the plan next time.
Autoscaling and the slow start
Autoscalers lag behind demand. After an outage, demand is pent up, which makes the lag worse. Consider raising the minimum capacity temporarily before the return and lowering it afterward. Check that new instances do not all start with empty caches and that startup work such as loading models or compiling does not delay them excessively.
Queues and batch jobs
Backlogs behave like retries. If a queue accumulated work during the outage, consumers may try to drain it all at once. Cap consumer concurrency, process in priority order and spread heavy jobs out. Our post on retry storms describes the pattern.
What good looks like in a drill
A good drill produces three numbers: how long the service took to reach normal latency, the peak database load during the return, and the share of requests rejected by load shedding. Compare them across drills. If they improve, the changes are working.
Try it yourself
Open the continuous monitoring demo and look at how metrics are tracked over time. A staged return depends on watching the right numbers at each step, so think about which three you would put on the screen.
Click through it at your own pace, or open it full screen.
Quick check
Is a restart the same as recovery?
No. A restart is when processes come back. Recovery is when users see normal speed and correctness. The gap between the two is where second outages tend to start.
Do I need a staged return for a small service?
Even small services benefit from a short ramp. The effort can be a single routing change and a checklist.
Is it ever right to bring everything back at once?
For small, stateless services with warm caches elsewhere it can be fine. Judge by the dependencies: if returning load can overwhelm something shared, stage it.
Sources
- Addressing Cascading Failures (Google SRE book)
- More details about the October 4 outage (Engineering at Meta, 5 October 2021)
Last reviewed: 19 October 2026.
