After an emergency fix applied only in production, treat the change as a loan: record exactly what changed, replay it in source control within a day or two, rebuild through the normal pipeline, and verify that prod and staging now match. Until then, your tested code is not your running code.
Why emergency fixes cause drift
Under pressure, someone upgrades a library on a live host or builds a one-off image. It works, the incident ends, and nobody updates the repository. Weeks later staging still holds the old package, the next deploy overwrites the fix, and the problem returns. Or the fix survives and nobody knows it exists.
The reconcile routine
- Capture the change while it is fresh. Record the package, old version, new version, host or image, who made it, and why. A chat message is not enough; use a ticket.
- Snapshot the production state after the change. An installed package list is enough.
- Replay the change in the repository: update the manifest and lockfile in a pull request that references the ticket.
- Test the change in staging with the normal checks.
- Rebuild and redeploy through the pipeline so production runs a built artifact, not a hand-edited one.
- Compare. Export installed packages from staging and production and confirm they match. See how to compare installed packages.
- Close the ticket only after step 6.
Set a deadline. Two business days is a reasonable starting point for a security fix; adjust to your release cadence.
Reduce the need for it
| Cause | Control |
|---|---|
| No fast path through the pipeline | Build an expedited lane that keeps scans and approvals but skips slow optional tests |
| Shell access on prod hosts | Restrict it and log it |
| Mutable servers | Move to rebuilt images |
| Nobody owns drift | Name an owner per service |
The expedited lane matters most. People bypass pipelines that take hours when a fix takes minutes. A lane that takes twenty minutes and still records an artifact removes the temptation.
Common mistakes
- Fixing production and forgetting that dev and staging remain vulnerable.
- Replaying the fix with a different version than the one running in prod.
- Rebuilding without a lockfile, which pulls in unrelated newer versions along with the fix.
- Closing the incident ticket without a drift check.
Try it yourself
Click through it at your own pace, or open it full screen.
See how a fix path is presented as a concrete version change. The same detail belongs in your reconcile ticket.
What Safeguard does here
Safeguard monitors dependencies against new advisories and can open fix pull requests, though autonomous remediation is off by default and every category starts on Manual, so a human decides. Try the free scan at gold.safeguard.sh/scan.
Quick check
Is it ever fine to patch prod by hand?
In a real emergency, yes, as long as it is recorded and reconciled quickly. The risk is leaving it undocumented.
Who should own the reconcile?
The person who made the change, with the service owner confirming the final comparison.
Last reviewed: 19 October 2026.
