A mirror runbook fits on one page and answers six questions: who owns it, how you know it is failing, who may switch builds to a fallback, what to tell developers, how to recover, and what to review afterwards. Write it before the first outage and test it once a quarter.
Why one page
During an outage nobody reads a 30 page document. They need names, thresholds and the next step. Keep it short enough to read in two minutes and store it where it is reachable when CI and chat are both degraded.
The template
1. Owner and backup
Name a team and a person, plus a backup. Include how to reach them outside normal channels. A mirror without a named owner turns into everyone's problem and no one's task.
2. What "failing" means
List the signals and thresholds. Use the five from what to monitor on an internal package mirror: hit ratio, upstream errors, authentication failures, metadata freshness and storage headroom. Add the synthetic check. Say which alert pages a human.
3. Who may trigger the fallback
Define who can authorize builds to use the fallback path, and under what conditions. A decision table helps.
| Condition | Action | Approver |
|---|---|---|
| Mirror unreachable, hotfix open | Use fallback for the hotfix pipeline | On-call lead |
| Mirror unreachable, no urgent release | Wait, notify | Mirror owner |
| Upstream registry down, mirror healthy | No action, announce | Mirror owner |
| Mirror serving a package under investigation | Block and purge | Security lead |
The reasoning behind fallbacks is in should CI fall back to the public registry.
4. What to tell developers
Prepare two short messages in advance: one for "the mirror is down, builds may fail, here is the workaround" and one for "recovered". Say what is affected, what to do and when you will update next. Keep wording factual. For customer facing wording see customer updates during an outage.
5. Recovery steps
List the commands or console steps to restart, restore from backup and re-point clients. Include where the backup lives and how long a restore takes, measured, not guessed. Include how to confirm recovery: the synthetic check passes, and a canary pipeline installs cleanly.
6. After the incident
Within a week, review: the timeline, how long builds were affected, whether fallback was used, and what changed in the packages that were pulled during it. The questions are in postmortem questions for your CI pipeline after a registry outage.
Add the facts people forget
- Where credentials for upstream and for clients are stored, and when they expire. See rotate a CI registry token without breaking builds.
- The list of ecosystems the mirror serves and the client configuration for each.
- How to purge a package fast after a compromise. See purge CI caches after a compromise.
Test the runbook
Once a quarter, walk through it with the on-call person using a tabletop scenario, or run a real exercise from game day plan for a security control outage. Fix every place where the page was unclear.
Where Safeguard fits
The runbook decides who acts. Safeguard's policies, gates and SBOM and dependency scanning give them the data: what the builds contained and whether a gate should block.
Quick check
Who should own the mirror?
A named team with a backup person, usually platform or developer tooling. Shared ownership with no name means slow response.
Where should the runbook live?
Somewhere reachable when CI and chat are degraded, such as a printed copy or a page hosted outside the failing system.
Last reviewed: 19 October 2026.
