Safeguard
Product•

Stop Carrier Base Images Drifting Behind Patches: A Platform Playbook

Inventory every base image by digest, rebuild on new fixes, gate production on KEV and severity, roll out in waves and keep a one-step rollback ready.

Safeguard Security Team
37 min read

A carrier platform team stops base images drifting behind patches by treating the rebuild as a loop, not a project. Know which images run where by digest, rebuild an image when a fix lands for something inside it, test it with the suite you already trust, block known exploited vulnerabilities from reaching production, and roll the patched image out in small waves with the previous digest kept for a one-step rollback.

This guide is for the platform or SRE team at a telecom operator that owns the shared base images, the build pipelines and the Kubernetes clusters that OSS, BSS, customer-facing APIs and, increasingly, network functions run on. The first two thirds work with whatever tools you have. The Safeguard walkthrough comes later, after an illustrative scenario that is a composite example, not a customer. Nothing here is legal or regulatory advice.

Key takeaways

  • An image is a frozen snapshot. NIST SP 800-190 says updates "must be made upstream in the images themselves" and then redeployed, so patching a running container is not the fix.
  • Drift is a tracking problem before it is a patching problem. If you cannot list every running image digest and its base, you cannot tell how far behind you are.
  • Put patched images through the same tests and production gate as any other change, and keep a human where your change process needs one.
  • Safeguard capabilities covered here, all GA: self-healing containers (GA), container image scanning in CI (GA), continuous scanning and vulnerability management with EPSS and KEV (GA), policies and gates with Block, Warn and Notify (GA), SBOM generation (GA), attestation (GA), autonomous remediation (GA, ships switched off with every category set to Manual), the local runner and CLI (GA), the mobile app (GA) and on-prem and air-gapped deployment (GA).

The situation: the job and who owns it

The job is simple to state. Every container image the carrier builds and runs should sit on a base image and a set of packages that are current enough that no known, fixable, serious vulnerability stays in production longer than the carrier's own policy allows.

The owner is usually the platform team. They publish the golden base images (a small set of operating system and language runtime images that every service builds FROM), they run the shared CI templates, and they run the clusters. Application teams own their own Dockerfiles and dependencies, but they inherit the base. When the base is stale, everyone is stale.

The trigger is quiet: a registry report, an auditor asking for patch currency, or an advisory for a library in every image. The platform lead then finds the bases were last rebuilt weeks ago, half the services pinned an old tag, and the rest deploy only when their owners ship a feature.

A bad week looks like this:

  1. Monday. A high severity fix lands for a library in the shared runtime base. The platform team rebuilds the base the same day.
  2. Tuesday. Nothing else moves. The rebuilt base sits in the registry. Forty services still run the old one because none of them shipped.
  3. Wednesday. The vulnerability is added to CISA's Known Exploited Vulnerabilities catalog. Security asks which production services are affected. Answering takes most of the day because the team has to map running pods back to image digests and digests back to bases.
  4. Thursday. The platform team opens forty pull requests by hand. Six fail tests for reasons unrelated to the patch. Two services belong to teams in a change freeze for a network maintenance window.
  5. Friday. On-call takes the pages for the rollouts that went wrong because readiness probes were never set. By the weekend the fleet is mostly patched, the team is tired, and nothing about the process is better for next time.

The cost is engineering time, on-call load and a crowded change calendar, not only risk. The fix is a loop that runs the same way every time.

Why carriers feel this more than most

Telecom platforms have a few traits that make base image drift worse:

  • Long-lived services. Billing mediation, provisioning, number portability and inventory systems can go months without a feature release. If deployment only follows features, the base never refreshes.
  • Strict change windows. Network-adjacent workloads often deploy only in overnight maintenance windows approved by a change board. Emergency patching has to fit a process that was built for planned work.
  • Mixed ownership. Some containers come from network equipment vendors as packaged network functions. The carrier runs them but does not build them, so the carrier cannot rebuild them either.
  • High scrutiny. Communications is one of CISA's critical infrastructure sectors, and the 2024 intrusions at carriers put patching discipline under a brighter light.

What the guidance says about stale images

No single US rule sets a patch clock for a carrier's container images. What exists is a set of primary sources that agree on the shape of a good process. Use them to justify the loop to your change board and your auditors.

NIST SP 800-190, Application Container Security Guide

NIST published SP 800-190 in September 2017, and it is still the most direct federal text on container risk.

  • Section 3.1.1, Image vulnerabilities. An image is a static archive. An image built with current components will, at some point, contain components with newly discovered vulnerabilities. With containers, NIST says the update happens in the image, which is then redeployed, rather than on the running host. A common risk, in NIST's framing, is running containers built from a vulnerable image version.
  • Section 4.1.1, countermeasures for image vulnerabilities. NIST asks for tools that cover the whole image lifecycle (build, registry, runtime), visibility into every layer (not only the base layer but frameworks and custom code), and policy-driven "quality gates" at each build and deploy stage so only images that meet policy progress. Its example is a rule that stops images with CVSS ratings above a chosen threshold.
  • Section 4.2.2, stale images in registries. NIST suggests pruning vulnerable images from registries and deploying by immutable, specific names. It warns that a "latest" tag is only a label and not a guarantee of freshness, so automation must make sure the tag it deploys really is the newest build.
  • Section 4.1.2. Containers should run in an immutable manner, and remote shell tools such as SSH should not run inside them. That is one more reason the patch goes into the image, not into the live container.

NIST SP 800-218, the Secure Software Development Framework

SSDF version 1.1 (February 2022) describes vulnerability response practices that map well onto image rebuilds:

  • PW.4.4 asks you to verify that acquired open source and other third-party components keep meeting your requirements throughout their life cycles. Its examples include regularly checking for publicly known vulnerabilities and building automatic detection into the toolchain. A base image is a third-party component.
  • RV.1.1 asks you to gather information on potential vulnerabilities in your software and its third-party components, for example by monitoring vulnerability databases.
  • RV.2.1 asks you to analyse each vulnerability enough to plan a response, including risk calculations based on exploitability and impact.
  • RV.2.2 asks you to plan and implement risk responses. Its examples cover a risk-based decision to remediate or accept, and a temporary mitigation when a permanent fix is not yet available.

CISA's guidance for communications infrastructure

In December 2024, CISA, NSA, FBI and partner agencies in Australia, Canada and New Zealand published "Enhanced Visibility and Hardening Guidance for Communications Infrastructure" in response to state-sponsored activity against telecom providers. The guide is written mostly about network devices, but two of its points apply directly to container platforms:

  • Keep the inventory of devices and firmware up to date so you can see and monitor what you run. For a container platform, that inventory is the list of running image digests and their bases.
  • Use a change management system that "anticipates both routine and emergency patching," and test and validate patches as part of it. That is a direct argument for a standing, tested rebuild loop instead of a scramble.

The guide also notes that the exploitation it describes lined up with existing weaknesses rather than novel techniques, which is the usual case for unpatched software.

The FCC context

In January 2025 the FCC issued a declaratory ruling that read section 105 of CALEA as requiring carriers to secure their networks. On 20 November 2025 the Commission rescinded that ruling in an Order on Reconsideration (FCC 25-81), withdrew the accompanying proposed rules, and pointed instead to commitments providers had made after the 2024 intrusions. Commissioner Gomez dissented. The practical point for a platform team is that there is no federal container patch deadline for carriers today, so your own written policy, your contracts and your customers' expectations set the clock. Write that clock down.

Exploitation signals: KEV and EPSS

Two free signals help you decide which rebuilds jump the queue:

  • CISA's KEV catalog is, in CISA's words, the authoritative source of vulnerabilities that have been exploited in the wild. A KEV-listed vulnerability in a production image is the clearest case for an emergency rebuild.
  • FIRST's EPSS estimates the probability that a published CVE will be exploited in the next 30 days, updated daily. Use it to rank everything that is not on KEV.
SourceWhat it asks forWhat it means for base images
NIST SP 800-190, 3.1.1Patch in the image and redeployRebuild and roll out; do not patch live containers
NIST SP 800-190, 4.1.1Lifecycle visibility, all layers, quality gatesScan base, runtime and app layers; gate each stage
NIST SP 800-190, 4.2.2Prune stale images, deploy immutable namesDeploy by digest or unique tag; retire old images
NIST SP 800-218, PW.4.4Keep verifying third-party components over their lifeTreat base images as dependencies that need ongoing checks
NIST SP 800-218, RV.2.2Plan and implement risk responses, with temporary mitigationsRebuild, defer with a dated exception, or mitigate
CISA communications guidance (Dec 2024)Up-to-date inventory; change process for routine and emergency patchesDigest inventory plus a standing emergency rebuild path
CISA KEV catalogPrioritise known exploited vulnerabilitiesKEV in production triggers the fast lane
FIRST EPSSProbability of exploitation in 30 daysRank the remaining rebuilds

How teams do it today and where it breaks

Most carrier platform teams already have some version of each piece below. The breaks come at the seams between them.

Nightly base image rebuilds

The platform team rebuilds the golden bases every night from upstream tags. This keeps the bases fresh, and it is the right first move. It breaks because a fresh base in the registry does nothing for a service until that service rebuilds and deploys. The drift moves from the base to every image built on it.

Pinning to stop surprises

Some services pin an exact tag or digest after a surprise breakage. Docker's own build guidance explains the trade-off: tags are mutable, so FROM alpine:3.21 resolves to the newest 3.21 patch at build time, while a digest pin gives a fully reproducible build but never moves unless someone changes it. Pinning is correct for reproducibility. It becomes drift when nothing updates the pin.

Rollouts that depend on feature releases

The biggest break is cultural. Deployments are tied to feature work and change tickets, so a patched image waits for a reason to ship. In a carrier with long-lived services and a change board, that wait can be weeks.

Step by step: a base image refresh loop with whatever tools you have

This is a loop your team can stand up with a registry, a CI system, a scanner and Kubernetes. Each step names what to produce, so you can check it is done.

1. Build a digest inventory

List every image running in every cluster, by digest, with its base image and owner. Kubernetes exposes the running image ID on each container status, so a short script against each cluster can produce the list. The output is a table like this one, which you can copy:

ServiceCluster or siteRunning image digestBase image and digestOwnerLast rebuiltCriticality
provisioning-apicore-eastsha256 digestruntime base, digestOSS teamdateHigh
billing-mediationcore-westsha256 digestruntime base, digestBSS teamdateHigh
portal-frontendedge-02sha256 digestweb base, digestDigital teamdateMedium

Without this table, every other step is guesswork. Refresh it on a schedule and after every rollout.

2. Decide your base image families and their owners

Keep the number of golden bases small (for example one per language runtime plus a minimal base). Write down who rebuilds each, how often, and which upstream tag or digest it follows. Fewer bases means fewer rebuild chains to track.

3. Set the patch clock in writing

Write a short policy that says how long a fixable vulnerability may stay in a production image, by class. A simple version:

ClassExample triggerTarget to stagingTarget to production
EmergencyKEV-listed, fix available, in a production imageSame dayNext approved emergency window
HighCritical or high severity, high EPSS, fix availableWithin daysNext routine window
RoutineEverything else with a fixNext scheduled rebuildNext routine window
No fix yetNo fixed version existsNot applicableDated exception with a mitigation

The numbers are yours to set. What matters is that the clock exists, that it treats KEV as the fast lane, and that "no fix yet" is handled by a dated exception rather than silence. That last line is SSDF RV.2.2 in practice.

4. Rebuild on a trigger, not only on a schedule

Rebuild an image when its base has a new digest or a fix lands for something inside it. Docker's guidance recommends --pull to fetch the newest base and --no-cache for a fully clean build; a rebuild that reuses a cached old base layer is not a patch. For pinned digests, have automation open a change that moves the pin.

5. Test the patched image with the suite you already trust

Run the service's normal test suite against the rebuilt image. Add a short smoke test for every image even when a service has no suite: does it start, does the health endpoint answer, does memory stay in range. A patch that changes a shared library can break startup in ways unit tests never see.

6. Gate before production

Put a gate in the pipeline that stops an image from being promoted when it carries a vulnerability your policy does not allow, for example a KEV-listed critical. This is the "quality gate" NIST SP 800-190 section 4.1.1 describes. Make the gate's exception path explicit: who can approve, for how long, with what justification.

7. Promote by digest and roll out in waves

Deploy the new image by digest or by a unique tag, never by reusing a tag the cluster may have cached. Use the Kubernetes Deployment rolling update settings to control pace. By default a Deployment keeps at least 75% of desired pods up (25% max unavailable) and allows up to 125% (25% max surge); carriers often tighten both for network-adjacent services. Set readiness probes so a pod that is not ready never takes traffic, and set a progress deadline so a stuck rollout is reported instead of hanging. Use PodDisruptionBudgets to protect availability during node drains, and remember that they govern voluntary disruptions such as drains, not every kind of pod replacement.

8. Keep a one-step rollback

Record the previous digest for every rollout. Kubernetes keeps revision history for Deployments, so kubectl rollout undo can return to the prior revision. Practise it. The fastest way to get a change board to approve emergency patching is to show that rollback is quick and boring.

9. Close the loop in the inventory

After rollout, update the digest inventory, retire the old image from the registry (NIST 4.2.2 again), and record the time from fix available to production for each class. That number is your own metric, not anyone else's benchmark.

A rollout checklist for change boards

Copy this into the emergency change template:

  • Image name, old digest, new digest
  • What changed (base digest, packages, lockfile diff)
  • Vulnerabilities closed, with KEV status and EPSS
  • Test results (suite and smoke test)
  • Gate result in staging
  • Rollout plan: waves, max unavailable, sites and order
  • Rollback command and owner
  • Exceptions still open for this image, with expiry dates

Pin by tag or by digest?

Both, at different stages. Build from a tag that follows the upstream patch stream so rebuilds pick up fixes, then deploy the resulting image by digest so what runs is exactly what was tested. The OCI image specification makes the digest the content identifier, so a digest is a fact while a tag is a pointer that can move. If you pin the base by digest, automate moving the pin.

How fast is fast enough for a carrier?

Your written policy, plus your contracts, sets the number. Most teams find that the bottleneck is not the rebuild, which is minutes of compute, but the wait for a deployment slot and a human review. That is why the loop above separates promotion to staging (which can be automatic) from promotion to production (which can wait for an approved window). Measure the gap between the two; it shows where your process, not your tooling, is slow.

What about distroless and scratch images?

They still need patching. The vulnerable parts arrive through the builder stage: the language runtime, statically linked libraries, or binaries copied in. Scan the final image and the build stage, keep an SBOM for the final image so you know which binaries are baked in, and rebuild when any of them gets a fix. A smaller image means fewer rebuilds, not none.

Where does a human stay in the loop?

Wherever your change process and your confidence say so. A sensible progression: first, automation opens a pull request against the Dockerfile or manifest repository and a person merges it; next, automation promotes to a staging tag and a person approves production in the next window; last, for stateless services with good tests and fast rollback, promotion runs to production with policy still able to hold a change. Move a service up only when its tests and rollback are proven. Network functions and anything touching lawful intercept or emergency calling usually stay at level one or two.

Illustrative scenario: ExampleTel's runtime base

This is a composite example, not a customer. ExampleTel is fictional, and every number below is an example input, not a result.

ExampleTel is a regional carrier. Its platform team of six engineers runs three Kubernetes clusters (two core regions and one cluster spread across edge sites) and publishes four golden bases: a minimal base, a JVM runtime, a Node.js runtime and a Python runtime. Example inputs: about 120 services build on those bases, about 30 of them are OSS and BSS systems that release less than once a quarter, and the change board meets twice a week with emergency approvals by phone.

The trigger

A fix lands upstream for a TLS library that ships in the minimal base and, through it, in the Node.js runtime base. ExampleTel's registry scanner shows the new CVE against the bases, and two days later it is added to the KEV catalog.

What the loop does

  1. The digest inventory answers the "which pods" question from a table, because every running digest is already mapped to its base.
  2. The patch clock classes the vulnerability as Emergency (KEV-listed, fix available, in production images).
  3. Automation rebuilds every image on the two affected bases with --pull, runs each service's tests plus the smoke test, and promotes passing images to a staging tag.
  4. The staging gate checks that no KEV-listed critical remains.
  5. The platform lead takes one emergency change to the board covering all passing images, with the rollout checklist attached.
  6. Rollout runs overnight in waves, one cluster at a time, with max unavailable tightened for the provisioning and billing services.
  7. Images whose tests failed get a dated exception with a mitigation note and an owner, and stay in the queue.
  8. The inventory updates, old images are retired, and the team records time from fix to production for the Emergency class.

Nothing in this scenario depends on a particular product. The next section shows how the same loop runs in Safeguard.

How Safeguard does this

Safeguard runs the loop above with less glue code. Each step below names the GA capability and links the docs page it comes from. Where the docs show a command or a screen, it is quoted as the docs show it; the snippets were copied from the docs, not run for this post.

1. Scan every image your pipeline builds, per environment

Container image scanning in CI (GA) adds one step after the image is built. The container images CI docs show it:

safeguard ci scan --image "ghcr.io/acme/api:$TAG" --environment prod --wait --fail-on critical

The scan runs on the CI runner, and only the results leave it. Safeguard records the image with one version per environment and tag (for example prod-1.4.2), so you can see what runs where, and the docs suggest failing only where it matters, such as --fail-on critical in production. Each container registry card under Integrations has a Set up CI/CD button that generates the steps for your CI.

That is the per-environment record the digest inventory needs, built from the pipeline. Software composition analysis (GA) covers container layers as well as application packages, and SBOM generation (GA) in SPDX and CycloneDX gives you the component list for each project.

2. Let continuous scanning catch new CVEs in images you already shipped

Continuous scanning and vulnerability management (GA) re-evaluate assets you have already scanned when the world changes. The continuous scanning docs describe rescans when vulnerability feeds change (new CVE, GHSA or OSV entries, KEV additions, EPSS changes) and when an asset changes, such as a new image tag in a registry. Findings carry severity, EPSS and KEV status, which is the ranking your patch clock needs.

3. Put a gate in front of production

Policies and gates (GA) express the quality gate from NIST SP 800-190 4.1.1. Per the Policies and Gates docs, policies evaluate SBOMs and produce findings, and gates enforce them. You create a policy under Settings, then Policies; conditions include severity, CVSS, EPSS and license.

Each policy has one Action, as the policy outcomes docs explain:

ActionWhat a failed policy does to the gate
BlockThe gate fails
WarnThe gate passes, with a warning naming the policy
NotifyThe gate passes, and a notification goes to the in-app bell, connected chat integrations and email

To fail a CI job on a gate, the docs show:

safeguard ci scan --wait --fail-on-gate <gate name or id | all>

It exits zero when the gate is Passed or Warning and non-zero when it is Failed, Error or Not evaluated. Policy outcomes are turned on per workspace by Safeguard, so check that the Outcome column appears on your Gates and Policies tables. When a team must ship despite a failing policy, approval runs through an exception with a business justification and an expiry date, decided by an approver and recorded in the audit trail. That is the dated exception from your patch clock.

A reasonable carrier setup is a Block policy on critical severity for the production gate, a Warn policy for highs, and a Notify policy on the production gate so new failures reach the platform channel. Use the KEV and EPSS fields on each finding to decide which failures go to the emergency lane of your patch clock.

4. Turn on self-healing containers, starting in advisory mode

Self-healing containers (GA) automate the rebuild loop. The self-healing containers docs describe four steps driven by continuous scanning:

  1. Detect. A new CVE affects a component in one of your container images.
  2. Plan. Griffin, Safeguard's remediation model, generates a patch plan.
  3. Rebuild. The runner rebuilds the image with the patch and runs your existing tests.
  4. Promote. If tests pass and the image passes guardrails, it is pushed to your registry under a new tag.

The docs describe three modes, which match the human-in-the-loop levels above:

ModeWhat Safeguard doesWhere people stay
AdvisoryDetects, plans and files a pull request against your Dockerfile repo or Kubernetes manifest repoPeople review and merge
Autonomous, promotion gatedDetects, plans, rebuilds and promotes to a staging tagA person approves promotion to production tags
Autonomous, continuousDetects, plans, rebuilds and promotes to productionA change that fails the production policy bundle is held for review

The docs recommend advisory mode for teams just turning self-healing on, and continuous mode only once you trust your test coverage. Separately, Safeguard's autonomous remediation ships switched off, with every category set to Manual, so nothing applies fixes on its own until your team turns it on and chooses how far it goes.

The docs show configuration as a file in the repository:

# .safeguard/self-heal.yaml
self_heal:
  mode: autonomous-staging   # advisory | autonomous-staging | continuous
  test_command: "npm test"
  notify:
    slack: "#sec-ops"
  strategy_overrides:
    - package: log4j-core
      prefer: upstream-bump

Or in the app: ESSCM, then Asset, then Self-Healing, then Configure.

5. Know which patch strategy each finding gets

The docs list the strategies Griffin picks per finding:

StrategyWhen it applies
Upstream bumpA patched upstream version exists
Backport patchUpstream has no patched release, but a fix is in the project's main line
Dependency pinA safe indirect dependency version exists but is not resolved; it is pinned
DeferNo safe fix exists; the finding stays open with a time-boxed exception

The Defer row is the "no fix yet" line of your patch clock, kept visible as an exception.

6. Base image and dependency-level healing

The docs cover both sources of drift. For base images, the common case is a FROM line whose base picks up a new CVE; self-healing rebuilds the image against the latest digest of that base. For FROM scratch and distroless images, the docs say Safeguard tracks which binaries are baked in and rebuilds the layer for the affected binary.

For packages pulled in during the build (npm ci, pip install, cargo fetch and the like), the docs describe resolving the lockfile to the minimum version that fixes the CVE, rebuilding the image, running the tests, and writing a lockfile diff into the image as a provenance attestation.

7. Tests before promotion

Self-healing calls your existing tests before it promotes. The docs list three harnesses: your CI provider (the build and test are delegated to your pipeline and Safeguard watches the result; the docs note there is no CLI command for this yet), the Safeguard local runner if you host it, and a synthetic smoke test (health endpoints, startup time, memory footprint) that runs on every rebuild even when no suite is declared.

The local runner and CLI (GA) are documented on the runner page. The runner is part of the safeguard CLI, polls the platform for jobs over outbound HTTPS, runs them locally and uploads only results, never source code. Headless use looks like this in the docs:

safeguard runner start --workspace ./path/to/repo
safeguard runner status
safeguard runner stop

8. What a healed image carries, and how to roll it back

The docs describe tagging conventions for healed images: <registry>/<image>:<original-tag>-sg-heal-<date> for staging promotions, and the original tag overwritten in continuous mode, with immutable digest pinning. Each healed image carries an SBOM attestation describing the patch, a plain-English explanation of what was fixed, and a signed provenance attestation that references the source commit and build environment. Attestation (GA) is described on the attestation and signing page.

Every heal is reversible, per the docs. The previous image digest is kept, a rollback is one click in the UI, and automatic rollback triggers on readiness probe failures, error-rate spikes, or policy violations found on the new image. The Self-Healing dashboard lists images being watched, heal cycles by state, a time-to-heal histogram and a rollback count, which feed the measures in the next section.

Rollout into your clusters stays with your own deployment tooling in this walkthrough: your rolling update settings, probes and change windows from steps 7 and 8 of the loop still apply.

9. Pair container healing with source fixes

Self-healing patches the image; the service's own declared dependencies are fixed in its repository by autonomous remediation (GA), which opens fix pull requests singly or in bulk, as the AI Remediate docs show. It ships switched off with every category set to Manual.

10. Run it where your network allows

Carriers with restricted or isolated environments can run Safeguard on-prem or air-gapped (GA). The air-gapped deployment docs list self-healing containers among the capabilities that run inside the perimeter, rebuilding with your mirrored registries, with vulnerability data delivered as signed snapshot bundles on a cadence you choose.

Requirement, what Safeguard does, what you still own

RequirementWhat Safeguard does (GA)What you still own
Inventory of images and bases (NIST 4.1.1, CISA guidance)Per-environment image versions from CI scans; SBOM generationMapping images to owners and criticality
Detect new vulnerabilities in shipped images (SSDF RV.1.1, PW.4.4)Continuous scanning with EPSS and KEVYour patch clock and its classes
Quality gates (NIST 4.1.1)Policies and gates with Block, Warn, Notify; exceptions with expiryChoosing thresholds; approving exceptions
Rebuild with the patch (NIST 3.1.1)Self-healing detect, plan, rebuild, promoteBase image families and upstream choices
Test before promotionCI, local runner or synthetic smoke testThe quality of your test suites
Roll out and roll backHealed image tags, preserved previous digest, rollbackRollout pace, probes, change windows
Temporary mitigation when no fix exists (SSDF RV.2.2)Defer strategy with a time-boxed exceptionThe mitigation itself and its owner

What to measure

Track these yourself to judge whether the loop works. They are measures, not promised outcomes.

  • Share of running images whose digest and base are in the inventory
  • Age of each golden base since its last rebuild
  • Number of production images carrying a KEV-listed vulnerability with a fix available, checked daily
  • Time from fix available to staging, and from staging to production, by patch clock class
  • The gap between staging and production promotion (this shows process delay, not tooling delay)
  • Rebuilds that failed tests, and why (patch broke it, or the test was flaky)
  • Rollbacks after a patched rollout, and their cause
  • Open exceptions, their age, and how many expired without action
  • Vendor network function images with open vulnerabilities, and the vendor's committed date

What Safeguard does not do

  • It does not rebuild images you do not build. Packaged network functions and other vendor images can be scanned and tracked, but the fix has to come from the vendor's release. Hold vendors to timelines in your contracts.
  • It does not admit or deny pods at your cluster boundary today. Safeguard's guardrails docs state that the Kubernetes admission controller is not published yet. Enforce in CI and at promotion, and keep your own cluster admission policies.
  • It does not decide your patch clock, your change windows or your exceptions. Safeguard records and enforces what you configure. The change board, the platform lead and the security team still decide.
  • It does not make weak tests strong. Self-healing promotes on your test results and a smoke test. If a service has thin coverage, keep it in advisory mode.
  • It does not turn autonomous fixing on for you. Autonomous remediation ships switched off with every category set to Manual, and self-healing starts where you configure it.
  • It does not replace your deployment tooling. Rolling update settings, probes, PodDisruptionBudgets and site order remain your configuration.
  • It does not give legal or regulatory advice. It produces records you can show; whether they satisfy a contract or a regulator is your call and your counsel's.

Common mistakes

  1. Rebuilding the base and stopping there. The base is fixed in the registry; production still runs the old one until each service rebuilds and deploys.
  2. Trusting the build cache. A rebuild that reuses the old base layer is not a patch. Use --pull and check the base digest in the result.
  3. Deploying by a reused tag. Nodes may keep a cached image for a tag. Deploy by digest or a unique tag.
  4. Skipping readiness probes. A patched image that fails to start takes traffic and pages on-call. Probes and rollback make emergency patching safe enough to approve.
  5. Treating every finding the same. Without KEV and EPSS ranking, teams batch everything and the urgent items wait with the rest.
  6. Letting "no fix yet" disappear. A finding with no fix needs a dated exception and a mitigation, or it is forgotten.

FAQ

How do I find which production pods run a vulnerable base image?

Map each running container's image digest to the image's base and its SBOM. If you scan images in CI per environment and keep an SBOM for each, the lookup is a search rather than a hunt. Without that mapping, you are reading pod specs by hand.

Should I rebuild every image nightly?

Nightly base rebuilds are good, but a service rebuild that never deploys does nothing for production. Rebuild on trigger (new base digest or new fix) and make sure rebuilds reach production on a clock.

Is patching a running container ever acceptable?

NIST SP 800-190 says updates belong in the image, which is then redeployed, and that containers should run immutably. Changing a running container leaves you with something that no longer matches any image you can test, sign or roll back to.

What should happen when no fix exists yet?

Record a time-boxed exception with a mitigation and an owner, as SSDF RV.2.2 suggests, and keep the finding open. In Safeguard this is the Defer strategy.

Does Safeguard deploy healed images into Kubernetes for me?

In this walkthrough rollout stays with your deployment tooling. Safeguard promotes the healed image to your registry under a new tag, keeps the previous digest for rollback, and the docs state the Kubernetes admission controller is not published yet.

Which self-healing mode should a carrier start with?

Advisory. Safeguard detects, plans and opens a pull request against your Dockerfile or manifest repository, and people merge it. Move a service to promotion-gated mode once its tests are reliable.

Can this run without outbound internet at edge sites?

Safeguard's air-gapped deployment docs list self-healing among the capabilities that run inside your perimeter, rebuilding with your mirrored registries and taking vulnerability data as signed snapshots.

Does the FCC require carriers to patch container images by a deadline?

Not today. The FCC rescinded its January 2025 CALEA declaratory ruling on 20 November 2025. Your own policy, contracts and customer commitments set the clock, so write it down.

The loop in this post keeps the images you build current. When the fix you need is not in any upstream release yet, or the vulnerable code sits in a dependency rather than the base, the options below cover what to do next.

Fix it without waiting for upstream

Whatever the vulnerability, most of the risk sits in two gaps. The first is the time between a fix being published and that fix actually running in your production builds. The second, and worse, is the time before any fix exists at all, while a maintainer works out a patch and a release schedule. Waiting through both gaps is a habit, not a requirement. Safeguard gives teams three ways to close them.

1. Autonomous remediation: pull requests, not tickets

When a fixed version exists, Safeguard's Griffin models open a tested, policy gated pull request that moves the affected dependency to the safe version. You can run it on one repository or in bulk across every project that carries the vulnerable version. Remediation categories include dependencies, licenses, security posture and findings, and each one is set separately. Reachability analysis ranks the vulnerable code paths your application actually calls first, so the first pull requests you merge are the ones that matter most. Autonomous remediation ships switched off, with every category set to Manual, so your team decides how much runs on its own and what still needs a human approval.

2. Gold Open Source: the safe version in seconds, free

gold.safeguard.sh is free and needs no login. Search any CVE or package to see affected and fixed versions, CVSS, EPSS, CISA KEV status and a recommended safe version, or use the "Am I affected?" checker. The same data is available through a free read API, README status badges and RSS threat feeds.

3. Gold Registry: start from components that are already fixed

Gold Registry provides hardened, drop in replacements for upstream packages and container images. Every Gold artifact is held continuously to zero exploitable critical or high CVEs and a completed license check, with signed build provenance (signed with Safeguard's key and verifiable with cosign, SLSA Build Level 3 as the design target) and a public evidence page you can hand to an auditor. An artifact loses its Gold badge if it regresses. Teams can point their own package directory at the registry so hardened equivalents resolve without changing how developers install anything.

When upstream has no fix yet, or will not ship one, Safeguard first works to get the fix accepted upstream. If the issue is severe or upstream will not move, Safeguard builds a fork, fixes the vulnerability there and publishes a hardened Gold build, so you are not stuck on a maintainer's release schedule. You can also request a hardened, compatibility tested build of a specific package. Ask which packages it covers before you plan around it.

Your situationWhat to use
A fixed version existsAutonomous remediation opens the upgrade pull request; Gold Open Source confirms the safe version
You need to know today whether you are affectedGold Open Source lookup and "Am I affected?" checker, free and without login
You want vulnerable versions to stop shippingSafeguard policies and gates in CI
Upstream has no fix yet, or will not ship oneA custom Gold request; fork and patch
You want to start clean on new projectsPull hardened components and images from Gold Registry

One honest limit: when the vulnerable code lives in a vendor appliance, a hosted service or an operating system runtime you do not build yourself, the fix still has to come from that vendor or distribution. The steps above apply to the open source dependencies and container images you build and ship, which is where most teams carry the largest and least tracked share of their exposure.

See how Safeguard would help

The first demo is a real CLI session. After install, safeguard opens the interactive terminal and is asked to "fix the lodash CVE in this repo". It reads the finding and the advisory, asks before any command that changes something, edits package.json to lodash 4.17.21, runs npm install and npm test with approval, prints a summary and pushes nothing, then shows /permissions, /model and /scan. It is the same pattern as a dependency-level heal: change the lockfile, run the tests, let a person approve.

Click through it at your own pace, or open it full screen.

The second demo sets up the runner on your own machine, approves it with its device code, and then scans from the terminal. Self-healing can run its tests on a self-hosted runner, so this is the piece that keeps rebuild tests inside your network.

Click through it at your own pace, or open it full screen.

The third demo is a real Android session: work is sent to a computer running the CLI, and the command is approved on the phone with Allow once, Allow for turn or Deny. For on-call at a carrier, that is how a person stays in the loop on a rebuild without being at a desk.

Click through it at your own pace, or open it full screen.

Quick check

Your platform team rebuilt the runtime base this morning with the fix. Is production patched?

No. The base is patched in the registry. Each service image built on it must be rebuilt, tested and rolled out, or production keeps running the old base. NIST SP 800-190 section 3.1.1 makes the same point: the update goes into the image, which is then redeployed.

In Safeguard, which self-healing mode opens a pull request for people to review and merge?

Advisory mode. Safeguard detects, plans and files a pull request against the Dockerfile or Kubernetes manifest repository. The docs recommend it for teams just turning self-healing on.

Sources

Related reading: telecommunications supply chain security, generating an SBOM from container images, Kubernetes supply chain security best practices, runtime versus static container analysis and container image supply chain incidents in 2026. For the short version of this job, see the deploy self-healing containers use case.

Try it yourself: download Safeguard or read the Safeguard docs.

Last reviewed: 25 October 2026. Product capabilities change; check docs.safeguard.sh for the current state.

Never miss an update

Weekly insights on software supply chain security, delivered to your inbox.

Self-healing security runs on Safeguard.

Your first fix PR is minutes away.

No sales call required, even your agent can complete the purchase over MCP.