Do not close an incident because the dashboards turned green. Check real user journeys, data integrity, backlogs, rotated secrets and any protections you switched off. Then watch for a defined period before standing down. Many second outages come from declaring victory too early.
Why green is not enough
Health checks test narrow things. A service can answer pings while checkout fails, or accept writes while a replica lags far behind. During an incident you also change things: scale up, disable alerts, relax rules, route around components. Recovery means undoing or deliberately keeping each of those changes.
The verification list
User journeys. Test the main paths by hand or with synthetic checks: sign in, the core action, payment, export. Try from outside your network.
Data integrity. Compare record counts or checksums against a known good source where you can. Look for gaps in the time window of the incident. Remember that replicated or restored data may be behind, as the guidance on replication versus backup explains.
Backlogs and queues. Work queued during the outage will arrive at once. Watch for retry storms, duplicate processing and downstream limits. Release backlogs gradually.
Replication and standby. Confirm the standby is healthy again and replication runs in the right direction. If you failed over, see the failback checklist.
Temporary changes. List every manual change made during the incident from the scribe's timeline: scaled capacity, feature flags, firewall rules, bypassed checks, emergency credentials. For each, decide to revert, keep, or ticket for review.
Secrets and access. If emergency access or credentials were used, rotate them. If a leak or compromise was suspected, confirm rotation finished everywhere that used the secret.
Monitoring and alerts. Re-enable anything silenced. Confirm alerts still fire, perhaps by triggering a test.
Security signals. If there was any chance of attack, review logs for the incident window: unusual logins, new tokens, unexpected changes. See outage or attack.
Customer side. Check support queues for new kinds of reports. Customers often find the leftovers.
Watch period
Pick a period before you stand down, based on the type of incident. Keep the incident lead or an on-call person watching, and keep rollback ready. If the metrics stay clean, declare the incident over and post the resolution message, as in customer updates during an outage.
Close properly
- Record the actual timeline and recovery time against your target.
- Schedule the review soon, while memory is fresh.
- Assign owners and dates to follow up actions.
Google's SRE guidance on postmortems calls for documenting impact, mitigation actions, root causes and follow up actions, and for a blameless approach that treats the review as a learning opportunity. It also lists review questions, including whether the impact assessment is complete and whether the action plan is appropriate with bug fixes prioritized correctly.
Update the runbook
Fix every step that was wrong or missing, within a week. See keeping runbooks true.
Quick check
How long should we watch after recovery?
Long enough to cover the next peak load or the next run of scheduled jobs, which varies. Decide it in advance in your runbook.
Who declares the incident over?
The incident lead, after checking the list with the people who ran the repair.
Sources
Last reviewed: 19 October 2026.
