On January 31, 2017, an engineer at GitLab ran a destructive command against what he believed was a lagging replica — it was the primary PostgreSQL server. Roughly 300GB of production data vanished in seconds: about 5,000 projects, 5,000 comments, and 700 user accounts from a six-hour window. The outage itself was recoverable in principle. What made it a five-alarm incident was what the team discovered while trying to restore: their scheduled pg_dump backups had been silently failing for weeks because of a version mismatch between the backup tool (9.2) and the live database (9.6), and two other backup mechanisms — Azure disk snapshots and S3 syncs — weren't working as anyone assumed either. Of five supposed backup layers, only one, an LVM snapshot taken about six hours earlier, was usable, and restoring it required roughly 18 hours of slow cross-datacenter copying. GitLab published the full postmortem, and it has since become one of the most-cited case studies in disaster recovery testing — not because the failure was unusual, but because untested backups are the norm, not the exception. This post lays out disaster recovery testing methods that would have caught GitLab's problem before it became an outage: tabletop exercises, failover drills, and RTO/RPO validation, scaled to how much a system actually matters.
What's the difference between RTO and RPO, and why do both need testing?
Recovery Time Objective (RTO) is the maximum tolerable downtime after a disruption — how long the business can function without the system. Recovery Point Objective (RPO) is the maximum tolerable data loss window — how much data, measured in time, you can afford to lose. Both are standard terms defined across NIST guidance, ISO 22301, and the well-architected frameworks published by AWS, Azure, and GCP, all of which map RTO/RPO pairs to specific DR architectures: backup-and-restore (cheapest, slowest, largest RPO), pilot light, warm standby, and multi-site active/active (fastest, most expensive, near-zero RPO). The reason both need active testing rather than paper sign-off is that each targets a different failure mode. An RPO target is only real if you've verified backups are complete, uncorrupted, and restorable — GitLab's backups technically "ran" for weeks while producing empty files. An RTO target is only real if you've timed an actual restore under realistic conditions, including the cross-datacenter transfer speeds that stretched GitLab's recovery to 18 hours.
What does NIST actually recommend for how often and how rigorously to test?
NIST SP 800-34 Rev. 1, the federal Contingency Planning Guide, lays out the main types of disaster recovery testing and ties test rigor to system impact level rather than prescribing one universal drill. Low-impact systems get a tabletop exercise — a discussion-based walkthrough where responders talk through a scenario without touching production. Moderate-impact systems require a functional exercise that validates specific recovery procedures against real (often non-production) systems. High-impact systems require a full-scale functional exercise that exercises the complete recovery process end-to-end, ideally including actual failover. The companion guide, NIST SP 800-84, covers how to design the test, training, and exercise program itself — scenario construction, roles, and after-action reporting. The through-line across both documents is annual testing at minimum, with additional tests triggered by significant infrastructure or architecture changes. ISO 22301, the international business continuity standard, states the same requirement in less prescriptive terms: continuity procedures must be exercised periodically and after any significant change to the environment they protect.
How should a tabletop exercise actually be run?
A tabletop exercise is a facilitated, discussion-only walkthrough of a specific failure scenario — no systems are touched. The value is in forcing named individuals to commit, out loud, to what they'd actually do: who declares the incident, who has authority to fail over a primary database, who notifies customers, and in what order. Effective tabletops use a written, specific scenario ("the primary region's database cluster is unreachable at 2:14 AM on a Saturday") rather than an abstract one, because vague scenarios produce vague answers. The facilitator should inject complications mid-exercise — a key person is unreachable, the documented runbook references a tool that was decommissioned six months ago — since real incidents rarely follow the clean path. GitLab's incident surfaced exactly this kind of gap: the runbook that led to the destructive command didn't clearly distinguish which server was primary, a labeling failure a tabletop walkthrough of "how do we fail over" could plausibly have exposed before it caused irreversible damage.
What makes a failover drill different from a tabletop, and what commonly breaks?
A failover drill is a functional or full-scale exercise where systems are actually moved — traffic is redirected to a standby region, a replica is promoted to primary, or a restore is executed against a real (non-production) target. This is where assumptions get falsified. The most common failure category is the one GitLab hit: backup or replication pipelines that appear healthy in monitoring but produce unusable output, whether from version mismatches, silent permission errors, or partial writes that never trigger an alert. The only way to catch this class of failure is a periodic test restore — pulling a backup and confirming the data it produces is complete and queryable, not just confirming the backup job exited with status zero. A second common failure is stale runbooks: documented failover steps that reference retired credentials, decommissioned hosts, or DNS records that were since repointed. A third is scope: teams drill the database layer but never test whether dependent services — caches, search indexes, background job queues — reconnect cleanly to a promoted primary, which is often where a "successful" failover quietly produces data inconsistency downstream.
How do you validate RTO/RPO numbers instead of just estimating them?
Validating RTO means timing a real restore, start to finish, under conditions that resemble a genuine incident — including the network paths, data volumes, and access constraints (someone paged at 2 AM, not a calm afternoon) that a real event would involve. GitLab's 18-hour recovery time was itself a form of RTO discovery: no one had measured how long a cross-datacenter LVM restore would actually take until they were forced to do it live. Validating RPO means confirming, after a timed restore, exactly how much data is missing relative to the point of failure, then comparing that gap against the documented target. A team claiming a 15-minute RPO on the strength of "backups run every 15 minutes" hasn't validated anything until a test restore confirms the backup taken at minute 15 actually contains minute-15 data. Both numbers should be re-validated whenever the underlying architecture changes — a database version upgrade, a new region, a change in backup tooling — since RTO/RPO figures measured against last year's infrastructure carry no guarantee against this year's.