Safeguard
Developer Tools•

CI is down and production is broken: can you still ship a hotfix?

When the build system is what failed, a hotfix needs another route. Plan a documented, logged emergency deploy path that keeps the checks you can't skip.

Safeguard Security Team
3 min read

Yes, but only if you planned a second route before you needed it. A sound emergency deploy path is documented, needs two people, keeps the checks that matter, and leaves a log. Improvised deploys from a laptop during an outage are how a bad day becomes a worse one.

Why this scenario is real

Your pipeline depends on a hosted service, a runner pool, a registry and credentials. Any of them can fail at the same time as production. If the only way to ship is through that pipeline, a pipeline outage during a production incident leaves you unable to fix anything.

Design the emergency path

Decide what you will allow. The point is a smaller, slower, safer route, not a free for all. Write down who can approve it and when it applies, for example "production is down or customers are losing data and the normal pipeline is unavailable".

Keep the artifact trusted. Build from a known commit on a clean machine or a separate runner, not from a developer's working directory with uncommitted changes. Record the commit hash that was built and deployed. Prefer redeploying an existing, already tested artifact, such as a previous release, over building something new.

Rollback first. The fastest hotfix is often not a fix. If the problem arrived with a recent release, redeploying the previous artifact removes it. That needs the old artifact to still exist in a store you can reach. See the rollback plan template.

Keep minimum checks. Decide which checks you will not skip: the build must pass, a smoke test must run, and a second person must review the diff. Skipping tests is a choice to record, not a default.

Use separate credentials. The emergency path needs deploy access that does not depend on the failed system. Store those credentials the way you store break glass accounts, sealed, alarmed and rotated after use. The post on break glass access covers the pattern.

Log everything. Write down who deployed what, when, from where, and why. Open a ticket afterwards to reconcile the emergency change back into the normal history, so the repository matches production.

A short procedure

  1. Incident lead declares the emergency path.
  2. Second approver confirms the need and reviews the change or the rollback target.
  3. Build or select the artifact, noting the commit.
  4. Deploy to the smallest scope first, if possible, such as one instance or a small share of traffic.
  5. Run the smoke test and watch metrics.
  6. Roll out the rest.
  7. Record the steps, then merge or cherry pick the change into the main branch and let the normal pipeline rebuild it once it is back.

What to test ahead of time

Run the drill once a quarter on a staging environment. Pretend the pipeline is unavailable. Time how long the emergency path takes and note missing access. A path that nobody has used since it was written probably has a stale credential or a missing step.

Risks to name

An emergency path bypasses controls such as required reviews, signing and policy checks. Keep it narrow, time limited and alarmed, and review every use afterwards. If it gets used for convenience, tighten access.

Quick check

Should we skip tests during an emergency deploy?

Avoid it. Run the smallest check that proves the fix or rollback works, and record any check you skip and why.

What is usually the fastest fix?

Redeploying the previous known good artifact, if the problem came with a recent release.

Last reviewed: 19 October 2026.

Never miss an update

Weekly insights on software supply chain security, delivered to your inbox.

Self-healing security runs on Safeguard.

Your first fix PR is minutes away.

No sales call required, even your agent can complete the purchase over MCP.