On-call runbook: a critical CVE in your production base image

When a critical CVE lands in your production base image at night, on-call has four jobs: confirm whether it is exploited and reachable from outside, find every image and running workload built on that base, decide between rebuilding now and mitigating until the day shift, and leave a handover note that lets someone else finish the job. You do not need to rebuild everything at 3 a.m. You need to know exactly what is exposed, fix the exposed part first, and write down the rest.

This post is a copyable runbook for that night. It works with whatever tools you have: a registry, a cluster, a CI system, a vulnerability feed. It covers the ten minute severity check, the inventory hunt, a decision table for rebuild now versus mitigate and wait, a safe rebuild and rollback sequence, how to prove the fix is running, and a handover template. Near the end we show how the same flow looks in Safeguard, and what Safeguard does not do for you.

Key takeaways

• Severity is not urgency. A CVSS 9.8 in a package no process loads is less urgent than a 7.5 on the CISA Known Exploited Vulnerabilities (KEV) list in an internet-facing service. Check KEV, EPSS and the distribution's advisory before deciding anything.
• The inventory question is two questions. Which images carry the vulnerable package, and which running workloads use those images by digest. Tags move; digests do not.
• A rebuild only helps when the distribution has shipped a fixed package. If it has not, your options are mitigation, a different base, or waiting, and the decision table below helps you pick.
• Rollback is a security decision too. Rolling back to the previous digest puts the vulnerable image back. Decide in advance when availability beats exposure.
• Verify by digest, not by deploy log. The fix is live when every pod's image ID matches the rebuilt digest and a fresh scan of that digest no longer shows the finding.
• Hand over in writing. The day shift needs what you know, what you changed, what is still exposed and what you decided not to do.
• In Safeguard: software composition analysis including container layers (GA), vulnerability management with CISA KEV and EPSS (GA), asset discovery across images and workloads (GA), policies and gates with Block, Warn and Notify actions (GA), Slack and Teams notifications (GA), the Safeguard CLI (GA), the mobile app (GA) and autonomous remediation (GA, ships switched off with every category set to Manual).

The situation

It is 02:40. The pager says something like "critical vulnerability in base image, 14 projects affected". You are the SRE or platform engineer on call at a US SaaS company. You did not write most of the services in that list. The security team is asleep or in another time zone. Somebody in a chat channel has already posted a link to a news story with the word "critical" in it.

You have three questions, and they are the right ones:

1. Which services are actually affected?
2. Do I rebuild everything now?
3. What do I hand to the day shift?

A bad night: you rebuild every image on the base, the CI queue backs up, one deploy breaks a health check, and you roll it back to the vulnerable image without writing it down. At 09:00 the day shift finds a half-patched fleet and no notes.

A good night: in ten minutes you know whether the CVE is exploited, in thirty you know which internet-facing services run the affected digest. You rebuild and roll those, verify by digest, set dated exceptions or mitigations for the rest, and write a handover a tired colleague can act on.

A worked example (illustrative)

An invented case with round numbers keeps the steps concrete.

• The CVE is a remote code execution flaw in a TLS library shipped as an OS package in your Debian-based base image. Call it CVE-YYYY-NNNNN.
• Your shared base image, base/debian-runtime, is used by 40 images in your registry.
• 22 of those images are deployed to production across 3 clusters, as Deployments, CronJobs and one DaemonSet.
• 5 services are internet-facing. The rest are internal APIs and workers.
• The distribution has published a fixed package for your release. Your base image has not been rebuilt since last Tuesday.

Swap in your own numbers; the decision structure stays the same.

The rules behind the runbook

Most SaaS companies are not bound by a federal directive, but these primary sources give you a defensible way to decide that reviewers recognise. This is not legal or audit advice.

NIST SP 800-61 Rev. 3: triage, prioritise, escalate, recover

NIST SP 800-61 Rev. 3, published in April 2025, recasts incident response as a Community Profile of the NIST Cybersecurity Framework 2.0. A critical CVE is a vulnerability, not yet an incident, but the response activities map cleanly:

• ID.RA-08 covers processes for receiving, analysing and responding to vulnerability disclosures. Your alert is one of those.
• RS.MA-02 says incident reports are triaged and validated: do a preliminary review, then estimate severity and the urgency needed to respond.
• RS.MA-03 asks you to prioritise based on scope, likely impact, time-critical nature and resource availability. "Resource availability" is the honest admission that one on-call engineer at 02:40 is a resource constraint.
• RS.MA-04 covers escalation (more people or more time) and elevation (involving management).
• RS.MA-01 suggests designating an incident lead for each incident. At night, that is you until you hand it over.
• RC.RP covers restoration: selecting recovery actions, verifying the integrity of recovered assets, declaring the end of recovery and completing documentation.

If you find any sign that the vulnerability has already been used against you, it stops being a patching task and becomes a security incident. Then your incident response plan, not this runbook, is in charge.

CISA KEV and BOD 26-04: exposure, exploitation, automation, impact

CISA's Known Exploited Vulnerabilities catalog lists vulnerabilities with evidence of exploitation in the wild. Each entry carries a date added, a due date, a required action and a field for known ransomware campaign use. CISA's own guidance is that organisations should use the catalog as an input to their vulnerability management prioritisation.

CISA's Binding Operational Directive 26-04, issued on 10 June 2026, replaced BOD 22-01 and BOD 19-02 for federal civilian agencies. It does not bind a private SaaS company, but its logic is the cleanest public model for a night-shift decision. The urgency of remediation comes from four questions:

1. Asset exposure. Is the vulnerable asset publicly exposed?
2. KEV status. Is the CVE in the KEV catalog?
3. Exploit automation. Can an adversary automate every step of exploitation?
4. Technical impact. Does exploitation give partial control or total control of the asset?

The directive's Table 1 turns those answers into remediation timelines. Two details matter for on-call. First, the shortest timelines pair remediation with a "forensic triage" of the asset, a check for whether it was already compromised. Second, the timelines are dynamic: removing a system from the internet is listed as a valid mitigation that changes "publicly exposed" from yes to no and pushes the deadline back, while a new KEV listing pulls it forward. That is the formal version of "mitigate now, fix properly in the morning". BOD 22-01 deadlines are history now; do not quote them as current.

FIRST EPSS: probability, not severity

The Exploit Prediction Scoring System, run by a special interest group at FIRST, estimates the probability that a vulnerability will see exploitation activity in the next 30 days. Scores are published daily for every CVE, as a probability between 0 and 1 and as a percentile. FIRST's FAQ gives the reading: across many CVEs scored around 0.05, about 5 in 100 would be expected to see exploitation activity over the next 30 days.

Two cautions: a day-old CVE's low score is weak evidence, and EPSS cannot tell you whether your service is reachable.

CVSS: a severity label, not a risk score

FIRST's CVSS v4.0 specification puts "Critical" at scores from 9.0 to 10.0. The score describes the flaw in the abstract; it does not know whether your process loads the library or whether the port is exposed. Use it to wake up, not to decide.

NIST SP 800-190: rebuild and redeploy, through gates

NIST SP 800-190, the Application Container Security Guide, has three passages that shape this runbook:

• Section 3.1.1 explains that images are static archives, so components go stale, and that unlike traditional servers, updates "must be made upstream in the images themselves," which are then redeployed.
• Section 4.1.1 recommends tools that cover the whole image lifecycle from build to registry to runtime, visibility into vulnerabilities at every layer (not only the base layer), and policy-driven "quality gates" that stop images over a chosen threshold from progressing.
• Section 4.2.2 warns about stale images in registries and recommends deploying immutable, versioned image names, noting that a "latest" tag is a label and not a guarantee of freshness.

OCI and Kubernetes: digests are the identity

The OCI Image Specification's content descriptor makes the digest a required property and says retrieved content should be verified against it. The Kubernetes documentation puts it plainly: digests are hashes of the image's content and are immutable, while tags can be moved to point to different images. Every inventory and verification step below works on digests for that reason.

Google SRE: one commander, a live document, a clear handoff

The incident management chapter of Google's Site Reliability Engineering book describes the roles that keep a night from turning into chaos. The incident commander holds the overall state and assigns work. Only the operations role should modify the system during an incident. The commander keeps a living incident document. And the handoff at the end of a shift has to be explicit, with the new commander confirming they have taken over. The book also argues for declaring incidents early: it is cheaper to close a small one than to spin up structure hours late.

Step by step: the runbook

The runbook has six stages. Times are targets for a single engineer, not service levels.

Open the incident document before stage 1. Write timestamps as you go. It costs one minute per step and saves an hour at handover.

How do you confirm severity and exploitation in ten minutes?

You are answering one question: how bad is this for us tonight? Use five checks, in this order, and stop collecting once the answer is clear.

1. Read the alert's CVE ID and package. Note the package name, the vulnerable version range and the base image name and tag. Copy them into the incident document. If the alert names only a CVE, find which package in your image matches.
2. Check KEV. Search the CVE ID in CISA's KEV catalog. If it is listed, note the date added and whether ransomware use is known. A KEV listing moves the decision towards "now" for anything exposed.
3. Check EPSS. Look up the score and percentile on FIRST's EPSS data. Record both and the date. Treat a high score as a push towards urgency; treat a low score on a day-old CVE with suspicion.
4. Read the distribution's advisory. This is the step most people skip. Your base image is built from a distribution, and the distribution decides whether and when a fixed package exists for your release. Debian's security tracker, for example, draws on Debian Security Advisories, the CVE list and NVD, and shows status per release. Note three things: is your release affected, is a fixed package published, and what version is it.
5. Read the upstream project's advisory. It usually says which feature or configuration is vulnerable. That tells you whether your services even use the vulnerable path, for example a server-side option you never enable.

Then write the one-line risk statement. Two examples:

"CVE-YYYY-NNNNN, TLS library RCE, in KEV since yesterday, EPSS high, fixed package available for our release, our base image is affected."

"CVE-YYYY-NNNNN, critical by CVSS, not in KEV, low EPSS, no fixed package for our release yet, vulnerable only when a server option we do not use is on."

The first is a "rebuild the exposed services tonight" night. The second is a "mitigate, gate and hand over" night.

Ten minute checklist

• CVE ID, package, vulnerable versions, fixed version (if any)
• KEV: listed or not, date added, ransomware flag
• EPSS: score, percentile, date
• Distribution status for your release: affected, fixed package published, version
• Upstream advisory: vulnerable feature or configuration
• Any sign of exploitation in your own logs or alerts (if yes, escalate to security now)

How do you find every image and running workload on the affected base?

This is where most nights go wrong, because "uses the base image" is not one list. You need three lists and a join.

List 1: images that contain the vulnerable package

Start from the package, not from the base image name. Teams often have more than one base: a runtime base, a build base, a slim variant, and a few images that pulled the distribution image directly. Any of them may carry the package.

Ways to build the list, from best to worst:

1. Query your SBOMs. If your pipeline produces an SBOM per image (CycloneDX or SPDX), search them for the package name and vulnerable version range. This is the fastest route when it exists.
2. Query your scanner's results. Most registry scanners can filter findings by CVE ID. Export the list of image repositories, tags and digests.
3. Use base image annotations. The OCI image specification defines org.opencontainers.image.base.name and org.opencontainers.image.base.digest. If your builds set them, you can list images by base digest. Many builds do not set them, so treat an empty result as "unknown", not "safe".
4. Search build files. Search every repository's Dockerfiles and build configs for FROM lines that reference the base. This finds images built from source you control, but it misses images built elsewhere and images that pin an old digest.

Write down digests, not just tags; a tag rebuilt twice this week points at only one of them.

List 2: running workloads and the digests they run

Ask the orchestrator what is actually running. In Kubernetes, the pod status records the resolved image ID for each container, which includes the digest. A read-only query like this lists namespace, pod and image ID across a cluster:

kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{range .status.containerStatuses[*]}{.imageID}{" "}{end}{"\n"}{end}'

Run it against every production cluster. Then remember the places this query does not show you:

• Init containers and ephemeral containers are in separate status fields.
• CronJobs and Jobs may have no running pod at 02:40. Their next run will pull whatever the spec says. Check the job templates, not just the pods.
• DaemonSets run on every node, including nodes that scale in later.
• Other runtimes. Serverless containers, managed container services, and plain VMs running containers all keep their own records.
• Staging and preview environments that share credentials or data with production.

List 3: exposure

For each running workload on an affected digest, record whether it is reachable from the internet, reachable only from inside, or not reachable at all (batch workers). Use your ingress, load balancer and service definitions, not memory. BOD 26-04's definition is a good test: accessible to unauthenticated or untrusted entities over public networks, regardless of where it physically sits.

The join

Combine the three lists into one table. This table is the most useful thing you will produce tonight.

In the illustrative case, the 40 images collapse to 22 production workloads, of which 5 are internet-facing and 1 is a DaemonSet on host networking. Those 6 are tonight's work. The other 16 are the day shift's.

When do you rebuild now, and when do you mitigate and wait?

Rebuilding is not free at night: it pulls in every change since that image's last build and deploys while the service's owners sleep. Mitigation is not free either: the vulnerable code stays, behind a control added under pressure.

Use this decision table per service group. Read it top to bottom and take the first row that matches.

Three questions break ties:

1. Is the pipeline healthy? If the last build of an image failed or has not run in weeks, a rebuild tonight is two changes at once. Prefer mitigation and hand it over.
2. Can you reach a service owner? For a critical, exposed service, a five minute call to the owner is worth more than a perfect runbook.
3. What does rollback cost? If the only rollback target is the vulnerable digest, a failed deploy puts you back where you started, plus an outage.

What counts as mitigation

Mitigation is a control that changes exposure or the vulnerable path without changing the image. Typical options:

• Remove or restrict internet exposure: tighten ingress rules, put the service behind an authenticated proxy, or shut a public endpoint for a few hours if the business can accept it.
• Disable the vulnerable feature through configuration, where the upstream advisory says that is enough.
• Add an edge rule that blocks the known request shape, if your edge supports it and the advisory gives you something precise to match. Treat it as a speed bump.
• Scale an exposed but non-critical service to zero until the morning.
• Suspend CronJobs that would start new pods on the vulnerable digest.

Write each mitigation down with how to undo it. A mitigation nobody remembers becomes an outage later.

How do you rebuild, promote and roll back safely?

The safe sequence is: rebuild the base, prove it is fixed, rebuild the exposed services on it, promote one at a time, and keep rollback honest.

Rebuild the base image

1. Confirm the fixed package exists for your distribution release, from the advisory you read in stage 1.
2. Rebuild the base with a fresh pull of the distribution image so you do not reuse a cached, vulnerable layer. With Docker, docker build --pull asks for a newer parent image, and --no-cache avoids reusing old layers:

docker build --pull --no-cache -t registry.example.com/base/debian-runtime:2026-10-25-hotfix .

3. If the distribution image itself has not been republished yet but the fixed package is in the distribution's security repository, a temporary package upgrade line in the base Dockerfile is a reasonable bridge. Mark it with a comment and a ticket so it is removed when the upstream image catches up.
4. Scan the new base digest with your scanner and confirm the CVE is gone and the package version matches the fixed version from the advisory.
5. Push with an immutable, dated tag and record the digest. Do not overwrite the old tag.

Rebuild the exposed services

1. Pin the new base by digest in each service's Dockerfile or build argument, so the rebuild uses exactly the image you scanned.
2. Build through the normal pipeline, not from a laptop, so the result has the usual tests, signatures and records.
3. Scan each new service digest. The CVE should be gone. If it is still present, the service may pull the package from somewhere else, such as a second stage or a vendored binary.

Promote one service at a time

Deploy to the smallest safe target first: a canary, one region or one cluster. Then watch the rollout complete:

kubectl rollout status deployment/checkout-api -n payments

Check error rates and latency for a few minutes before moving on. Kubernetes keeps rollout history for Deployments by default, which is what makes the rollback step possible:

kubectl rollout history deployment/checkout-api -n payments

Roll back with eyes open

If the new version breaks, the documented rollback is:

kubectl rollout undo deployment/checkout-api -n payments

That puts the previous image back, which is the vulnerable digest. Decide before you deploy which way you will lean:

• For an internet-facing service with a KEV-listed CVE, prefer a fix forward (a quick patch or a configuration change) or a mitigation in front of the rolled back version.
• For an internal service, rolling back and handing over is usually fine.

Either way, write the rollback in the incident document with the time and the digest it restored. An unrecorded rollback is the most common way a fleet ends up "patched" on paper and vulnerable in practice.

How do you verify the fix is actually running?

A green deploy is not proof. Verification has three parts.

1. Digest check. Re-run the pod query from stage 2 against every cluster. Every container in the "now" group should report the new digest. Look for stragglers: pods stuck terminating, a second replica set that did not scale down, a node that failed to pull and fell back to a cached image.
2. Fresh scan of the running digest. Scan the exact digest you see in the pod status, not the tag you meant to deploy. The finding should be gone and the package version should match the fixed version.
3. Package check, where the image has a shell. For images with a package manager, a read-only query inside one pod confirms the installed version, for example dpkg-query -W on Debian-based images, rpm -q on RPM-based images, or apk info -v on Alpine. Minimal images often have no shell; rely on the scan of the digest instead.

Then check the places that do not show up as running pods:

• CronJob and Job templates now reference the new digest.
• Autoscaling groups and node pools will not pull the old base for a DaemonSet when they scale.
• The old tags in the registry are either marked as vulnerable, pruned per NIST SP 800-190 section 4.2.2, or blocked by a gate so nobody redeploys them tomorrow.

What goes in the handover note?

The handover is where the night either pays off or gets wasted. The Google SRE book's advice is simple: hand over command explicitly, and make sure the next person confirms they have it. Put the note in the incident document and link it in the channel.

Copy this template:

HANDOVER: CVE-YYYY-NNNNN in base/debian-runtime
Shift: 02:40 to 07:30 (on-call: <name>)
Status: Partially remediated. 6 of 22 production workloads fixed. No sign of exploitation found.

1. Risk statement
   KEV: yes, added <date>. EPSS: <score> (<percentile>) on <date>.
   Distribution fix: <package> <fixed version> published for our release.
   Vulnerable path: <feature/configuration from upstream advisory>.

2. What is fixed (verified by digest and scan)
   base/debian-runtime -> sha256:<new>, tag 2026-10-25-hotfix
   node-agent, checkout-api, search-api, <x>, <y>, <z>: new digests, scan clean.

3. What is still exposed
   16 internal workloads on sha256:<old> (list in table above).
   2 CronJobs suspended: nightly-export, <job>. Resume after rebuild.

4. Mitigations in place (and how to undo them)
   <ingress change>, applied 03:55, undo: <command or PR>.

5. Rollbacks
   None. / <service> rolled back at <time> to sha256:<old> because <reason>.

6. Gates and exceptions
   Gate <name> now blocks new builds on the old base.
   Exception for <service> expires <date>, owner <team>.

7. Decisions I did not make
   Whether to rebuild internal services before or after the release freeze.
   Whether to notify any customers (not my call).

8. Next actions for day shift
   Rebuild internal workloads in this order: ...
   Remove temporary package upgrade line once upstream base is republished.
   Close incident when all 22 workloads verified.

Handover accepted by: <name> at <time>

Keep item 7. On-call engineers should not make customer notification, contractual or regulatory calls alone at night, and the note should say so plainly.

Common mistakes

• Starting with the rebuild. Rebuilding before you know what is exposed spends the night on low-risk services and leaves the exposed ones for last.
• Trusting tags. A tag tells you what someone intended. A digest tells you what is running.
• Forgetting scheduled work. CronJobs and Jobs will start fresh pods on the old digest at their next run.
• Ignoring the distribution's status. Rebuilding against a distribution that has not shipped a fix yet produces a new digest with the same vulnerability and a false sense of progress.
• Patching inside running containers. NIST SP 800-190 is clear that container fixes happen in the image, which is then redeployed. In-place changes vanish on restart and the registry no longer describes reality. Retagging an old image as the fixed version is the same mistake.
• Rolling back silently. A rollback to the vulnerable digest without a note is how fleets end up half fixed.
• Paging everyone. A critical CVE in an internal-only batch worker rarely justifies waking three teams. Use the decision table.
• No dated exceptions. "We will get to it" with no expiry date becomes permanent exposure.
• Closing on deploy, not verification. The incident ends when digests and scans agree, not when the pipeline turns green.

How Safeguard does this

Everything above works with any tooling. This section shows how the same runbook looks when Safeguard is connected to your registries, clusters and notification channels. Commands and screens are only those shown in Safeguard's docs; where the docs do not show a screen, we describe the outcome instead.

1. The alert reaches you in Slack or Teams. Notifications to Slack, Microsoft Teams, Google Chat and Discord are GA. In Safeguard, you connect Slack from Integrations, on the Notification tab, with either a bot token or an incoming webhook for a single channel. Safeguard posts when the events you subscribed to happen, for example a scan finishing or a new critical finding. It only posts and lists channels; it does not read messages. Point it at the channel your on-call rota already watches.

2. The finding is already re-evaluated. Vulnerability management is GA. Safeguard's docs say it performs daily continuous scanning of your projects and re-evaluates them when new CVEs are published. Findings fall into Critical, High, Medium, Low and an Exploitable category for vulnerabilities with known active exploitation. Data sources include NVD, GitHub Security Advisories, OSV, CISA KEV and vendor security bulletins.

3. Run the ten minute check from one screen. On the Vulnerabilities page, each finding shows the CVE ID, CVSS score, EPSS score, whether it is actively exploited, the affected component and version, the fixed version, references to advisories and a reachability status. You can filter by severity, EPSS range, exploitable status, component name and fix availability. The docs also show natural language search, for example asking for critical vulnerabilities in production that are actively exploited. Reachability analysis and prioritisation (GA) folds EPSS, KEV, reachability and business context such as exposure level into a priority. Read the references to the distribution advisory yourself; the fixed version tells you whether a rebuild can help.

4. Find the affected images. Software composition analysis is GA and covers container layers as well as language packages. Connect your registry through its integration card; for any registry that speaks the OCI distribution API, the OCI Registry card lists repositories and tags, and each image you select becomes a project that Safeguard scans for an SBOM and findings. The docs are clear that Safeguard only reads and never pushes, deletes or retags images. The card has no webhooks, so new tags are scanned on the schedule you set under Integrations, Scheduled, or when you rescan by hand. At night, rescan by hand before you trust the list.

5. Find the running workloads. Asset discovery is GA. Safeguard's docs describe discovered assets including container images and runtime workloads, tagged with an owner and an environment such as production or staging, and an asset graph that links a container image to the Kubernetes workload it runs in. The docs give the kind of question you would ask tonight, such as which production workloads contain a given package version, answered from the Search page in the platform. Runtime coverage depends on what you have connected; a cluster that is not connected is not in the answer.

6. Scan a candidate image from your terminal. The Safeguard CLI is GA. Sign in with safeguard login (or safeguard login --no-browser on a jump host), refresh the vulnerability databases, and scan the rebuilt base or service image:

safeguard update
safeguard scan --image myregistry.azurecr.io/myapp:v1 \
  --registry-username "$REGISTRY_USER" \
  --registry-password "$REGISTRY_PASSWORD"

Results are written as JSON under ./output, including vuln.json. The docs note that safeguard scan exits 0 when the scan ran, whatever it found, and 2 when results were written but the vulnerability scan did not actually run, so a script never mistakes that for a clean pass. There is no severity threshold flag; gating is done by policy gates in the platform. To see results next to your other projects, scan from the interactive terminal with /scan . while signed in. If your hosts run Podman, the docs show safeguard podman inventory to list running containers grouped by image digest, and safeguard podman scan --running to scan the images of running containers, with results kept on the machine.

7. Stop the old digest from shipping again. Policies and gates are GA. A policy such as the "No Critical Vulnerabilities" template generates a finding; a gate enforces it. Each policy's action is Block (the gate fails), Warn (the gate passes with a warning) or Notify (the gate passes and a notification is sent). With policy outcomes on for your workspace, the documented CI step fails a job when the gate fails:

safeguard ci scan --wait --fail-on-gate <gate>

Gates are evaluated by the platform on the scans it runs for your connected repositories. When a service genuinely cannot be fixed tonight, request an exception: pick the policy and finding, give a business justification, set an expiration date and submit it for approval. Approvers are notified, and the decision is recorded in an audit trail. On the Vulnerabilities page, Accept Risk works the same way, with an expiry date and a justification. These are the dated exceptions your handover note refers to.

8. Let the fix flow where a fix is a dependency change. Autonomous remediation is GA and ships switched off, with every category set to Manual. For language dependencies inside your services, AI Remediate on the Vulnerabilities page can generate remediation plans and pull requests for review. For an OS package in a base image, the fix is still a rebuild through your pipeline. The Safeguard CLI's interactive assistant can help edit a Dockerfile or build file in a repository; it asks before any command that changes something.

9. Approve from your phone. The Safeguard mobile app (GA, iOS and Android) can send work to a computer running the CLI and lets you approve each command with Allow once, Allow for turn or Deny. That helps when you have stepped away from the laptop between deploy waves.

10. Verify and record. Rescan the rebuilt image projects and confirm the finding is gone for the new digests. The Mitigations area in the platform tracks mitigations and risk acceptances, which gives the day shift one place to see what is still open and when each exception expires.

Requirement, Safeguard, and what you still own

What Safeguard does not do

• It does not rebuild or redeploy your workloads. Safeguard reads registries and never pushes, deletes or retags images. Building the new base, rolling it out, restarting pods and rolling back are your pipeline's and orchestrator's jobs.
• It does not change your orchestrator or your edge. Ingress rules, network policies, edge rules, scaling to zero and suspending CronJobs are changes you make in your own systems.
• It cannot make a fix exist. A rebuild only helps when the distribution has shipped a fixed package. Until then, Safeguard can show you the exposure, gate new builds and record a dated exception, but the package fix comes from the distribution.
• Runtime coverage depends on what is connected. A cluster, account or registry that is not connected is not in the asset graph or the findings. The OCI Registry card has no webhooks, so new tags wait for the schedule or a manual rescan, and the docs note that private image pulls during the scan do not yet use the card's credential.
• There is no CLI command for querying the asset graph yet. The docs say so; use the Search page in the platform.
• It does not do forensics. If you suspect exploitation, evidence preservation and investigation follow your incident response plan and your own tooling.
• On-call judgement stays human. Whether to rebuild now, roll back, notify customers or wake an owner is your decision and your company's. Safeguard's prioritisation is an input to that decision, not a substitute for it.
• The REST API on some docs pages is not live yet. Several Safeguard docs pages carry a banner that the /v1 endpoints are not yet available. Use the platform, the CLI and CI integration instead.

FAQ

What should on-call do first when a critical CVE hits a production base image?

Confirm before you act. Check whether the CVE is in CISA's KEV catalog, look up its EPSS score, and read your distribution's advisory to see whether a fixed package exists for your release. Write a one-line risk statement. Only then build the list of affected workloads.

Do I need to rebuild every image that uses the base image tonight?

Usually not. Rebuild exposed and widely privileged workloads first, gate new builds on the old base, and hand the rest over with dated exceptions.

What if the distribution has not released a fix yet?

A rebuild will not help. Reduce exposure (restrict ingress, disable the vulnerable feature, block at the edge), consider a different base image if one is fixed, record a dated exception, and watch the distribution's tracker.

How do I find which Kubernetes pods are running a vulnerable image?

Query each pod's container image IDs, which include the digest, in every cluster. Check init containers, job templates and non-Kubernetes runtimes separately.

Is it safe to patch packages inside a running container?

No. NIST SP 800-190 says container updates are made in the image and then redeployed. In-place changes disappear when the pod restarts and leave your registry describing something that is not running.

What does rollback do to my vulnerability exposure?

It restores the previous, vulnerable image. Decide in advance whether to fix forward or mitigate in front of it, and record every rollback.

How do I prove the fix is actually running?

Compare the digest in every pod's status with the rebuilt digest, scan that exact running digest, and where the image has a package manager, confirm the installed version. Check job templates and node pools too.

Is a critical CVE in a base image a security incident?

Not by itself. It is a vulnerability that needs a response. It becomes a security incident when you find signs it was used against you, and then your incident response plan takes over, including forensic triage before you redeploy.

Does BOD 26-04 apply to a private SaaS company?

It binds federal civilian executive branch agencies, not private companies, unless a contract says otherwise. Its four factors (exposure, KEV status, exploit automation, technical impact) are still a useful, public model for night-shift decisions.

What should a CVE handover note include?

The risk statement, what is fixed and verified by digest, what is still exposed, mitigations and how to undo them, any rollbacks, gates and exceptions with expiry dates, decisions you deliberately left for others, and the next actions. End with who accepted the handover and when.

Related reading

• The first hour after a platform flags a critical dependency flaw
• How to route security alerts by on-call hours and severity
• How many pages per on-call shift is too many
• Container vulnerability SLA by severity and fix availability

The runbook above deals with the night a fix exists and has to reach production, or the night it does not exist yet and you have to hold the line. The section below covers both gaps for the open source dependencies and images you build, and its closing limit applies directly here: an OS package fix still comes from your distribution.

Fix it without waiting for upstream

Whatever the vulnerability, most of the risk sits in two gaps. The first is the time between a fix being published and that fix actually running in your production builds. The second, and worse, is the time before any fix exists at all, while a maintainer works out a patch and a release schedule. Waiting through both gaps is a habit, not a requirement. Safeguard gives teams three ways to close them.

1. Autonomous remediation: pull requests, not tickets

When a fixed version exists, Safeguard's Griffin models open a tested, policy gated pull request that moves the affected dependency to the safe version. You can run it on one repository or in bulk across every project that carries the vulnerable version. Remediation categories include dependencies, licenses, security posture and findings, and each one is set separately. Reachability analysis ranks the vulnerable code paths your application actually calls first, so the first pull requests you merge are the ones that matter most. Autonomous remediation ships switched off, with every category set to Manual, so your team decides how much runs on its own and what still needs a human approval.

2. Gold Open Source: the safe version in seconds, free

gold.safeguard.sh is free and needs no login. Search any CVE or package to see affected and fixed versions, CVSS, EPSS, CISA KEV status and a recommended safe version, or use the "Am I affected?" checker. The same data is available through a free read API, README status badges and RSS threat feeds.

3. Gold Registry: start from components that are already fixed

Gold Registry provides hardened, drop in replacements for upstream packages and container images. Every Gold artifact is held continuously to zero exploitable critical or high CVEs and a completed license check, with signed build provenance (signed with Safeguard's key and verifiable with cosign, SLSA Build Level 3 as the design target) and a public evidence page you can hand to an auditor. An artifact loses its Gold badge if it regresses. Teams can point their own package directory at the registry so hardened equivalents resolve without changing how developers install anything.

When upstream has no fix yet, or will not ship one, Safeguard first works to get the fix accepted upstream. If the issue is severe or upstream will not move, Safeguard builds a fork, fixes the vulnerability there and publishes a hardened Gold build, so you are not stuck on a maintainer's release schedule. You can also request a hardened, compatibility tested build of a specific package. Ask which packages it covers before you plan around it.

One honest limit: when the vulnerable code lives in a vendor appliance, a hosted service or an operating system runtime you do not build yourself, the fix still has to come from that vendor or distribution. The steps above apply to the open source dependencies and container images you build and ship, which is where most teams carry the largest and least tracked share of their exposure.

See how Safeguard would help

The first demo is a real Safeguard CLI session. It installs the CLI, opens the interactive terminal, asks it to fix a lodash CVE in a repository, and shows it reading the finding and advisory, asking before any command that changes something, editing the dependency, running install and tests with approval, and pushing nothing. On a base image night, the same terminal is where you sign in, run /scan . on the repository that holds your Dockerfile, and scan rebuilt images before you promote them.

Click through it at your own pace, or open it full screen.

The second demo is a real Android session with the Safeguard mobile app. It sends work to a computer running the CLI and approves the command on the phone with Allow once, Allow for turn or Deny. For on-call, that means you can keep a rebuild and scan moving from your phone while you wait between deploy waves, without giving the assistant blanket permission.

Click through it at your own pace, or open it full screen.

To try it yourself, download the apps and CLI at safeguard.sh/download and read the docs at docs.safeguard.sh.

Quick check

Sources

• NIST SP 800-61 Rev. 3, Incident Response Recommendations and Considerations for Cybersecurity Risk Management (CSRC)
• NIST SP 800-61 Rev. 3, full text (PDF)
• NIST SP 800-190, Application Container Security Guide (CSRC)
• NIST SP 800-190, full text (PDF)
• CISA Known Exploited Vulnerabilities Catalog
• CISA BOD 26-04: Prioritizing Security Updates Based on Risk
• FIRST, Exploit Prediction Scoring System (EPSS)
• FIRST, EPSS frequently asked questions
• FIRST, CVSS v4.0 Specification Document
• OCI Image Specification, content descriptors
• OCI Image Specification, pre-defined annotation keys
• Kubernetes documentation, Images
• Kubernetes documentation, Deployments
• Google, Site Reliability Engineering, Managing Incidents
• Debian Security Bug Tracker
• Safeguard Docs, Vulnerabilities
• Safeguard Docs, Policies and Gates
• Safeguard Docs, OCI Registry
• Safeguard Docs, Slack
• Safeguard Docs, CLI Usage
• Safeguard Docs, Asset Discovery

Last reviewed: 25 October 2026. Product capabilities change; check docs.safeguard.sh for the current state.