Your pipeline is part of your production system. If it is down or cannot be trusted, you still need a safe way to ship a fix. Keep pipeline definitions in code, document a manual deploy path with approvals, and plan secret rotation for the compromised case.
Two different failures
Outage. The CI service or runners are unavailable. The code is fine; you cannot build or deploy through the normal route.
Compromise. Someone changed a workflow, a runner or a secret. Artifacts built recently may not be trustworthy, and credentials in the pipeline are burned.
The first needs speed. The second needs care. Your plan should say how to tell them apart and who decides.
What to keep outside the pipeline
- Pipeline definitions in the repository, reviewed like code.
- Build instructions that work on a clean machine, so a laptop or a spare runner can reproduce an artifact.
- A list of every secret and cloud role the pipeline uses, with an owner for each.
- Copies of recent release artifacts and their checksums.
- Contact details for the people who can approve an emergency deploy.
If the only way to build is the CI service, a CI outage is a deploy outage. See also planning for a security control outage.
A manual deploy path that is still safe
Write down an emergency route and rehearse it:
- Build from a tagged, reviewed commit on a clean machine.
- Record the artifact checksum.
- Require a second person to approve the deploy.
- Use a short-lived, narrowly scoped credential issued for this event.
- Log everything and review it afterward.
Do not keep a standing admin key "for emergencies." Short-lived access, as in OIDC instead of long-lived cloud secrets, limits what a stolen credential can do.
If you suspect compromise
Treat it as an incident. Stop new deploys, preserve logs, and identify the last known good commit and artifact. Rotate every secret the pipeline could read, and rebuild runners from a clean image rather than cleaning them. Rebuild release artifacts from source and compare them with what you shipped. Prior incidents show why this matters; for one maintainer-side example, read if your npm package was compromised.
After recovery
Add the gaps you found to the plan, and run the manual path again within a quarter. Gates in the pipeline can help prevent a repeat. Safeguard offers policies and gates for pull requests and builds, plus dependency scanning, and a free scan at https://gold.safeguard.sh/scan.
Quick check
Can we skip approvals during an outage?
Avoid it. Speed matters, but a second approver and a short-lived credential cost minutes and protect you from a bad emergency deploy.
What is the first step if the pipeline may be compromised?
Stop deploys and preserve logs, then rotate secrets the pipeline could read and rebuild from a known good commit.
Last reviewed: 19 October 2026.
