Safeguard
Cloud Security

Connecting build, deploy, and runtime security into one AppSec lifecycle

The XZ Utils backdoor (CVE-2024-3094) was found in the build chain; most tools that would have caught it stop at deploy. Here's how to close that gap.

Safeguard Research Team
Research
7 min read

On March 29, 2024, Microsoft engineer Andres Freund noticed SSH logins on a Debian testing box taking a few hundred milliseconds longer than they should have — and traced it to a backdoor planted inside XZ Utils, a compression library linked into sshd on most major Linux distributions. The attack, tracked as CVE-2024-3094, was not a coding mistake; it was a multi-year social-engineering campaign that compromised the library's build process itself, hiding malicious code in test artifacts that only activated during the linking stage. Three years earlier, Log4Shell (CVE-2021-44228) showed the opposite failure mode: a single transitive Java logging dependency, present in countless build artifacts, reached production with a remote-code-execution flaw and detonated at runtime. Most organizations still run build scanning, deploy-time signature checks, and runtime detection as three unconnected tools bought from three different vendors at three different times — which means a finding that starts in a pom.xml never talks to the admission controller, and neither talks to the eBPF sensor watching the pod at 3 a.m. This piece lays out what a genuinely connected build-to-runtime lifecycle looks like and why each handoff matters.

What does build-time security actually need to produce?

Build-time security needs to produce evidence, not just a pass/fail scan result — specifically, a record of what was built, from what source, by what pipeline, and with what components. The OpenSSF's SLSA (Supply-chain Levels for Software Artifacts) framework formalizes this in its current v1.0 spec as three build levels: L1 requires only that a provenance document exists; L2 requires that provenance be generated and signed by a hosted build platform rather than a user-controlled script; and L3 requires build platform controls that isolate runs from each other and keep signing material out of reach of the build steps themselves, making forged provenance far harder to produce. A future L4, covering hermetic and reproducible builds, remains aspirational and has not shipped as a production level. Executive Order 14028, signed May 12, 2021 in the wake of the SolarWinds compromise, pushed a parallel requirement into federal procurement — SBOM generation and secure development attestations from software suppliers — and that mandate has since become a de facto baseline for commercial software buyers too. The XZ backdoor is the sharpest argument for why provenance matters more than a clean scan: the malicious code was inserted into release tarballs that differed from the public git history, something a dependency-CVE scan would never catch, but a build system generating SLSA provenance tied to a specific, reviewable source commit would surface immediately.

How does deploy-time verification close the gap between a build and a running workload?

Deploy-time verification closes that gap by checking, at the moment an artifact is about to run, that its signature, SBOM, and attestations actually match what the build system produced — rather than trusting that nothing changed between build and deploy. Sigstore's Fulcio and Rekor components made this practical at scale: Fulcio issues short-lived certificates bound to a workload's OIDC identity (a GitHub Actions run, a GitLab CI job) instead of requiring teams to manage long-lived signing keys, and Rekor records every signature in a public, append-only transparency log that anyone can audit. This "keyless signing" pattern is now the basis for npm's package provenance feature and is used by major container registries. The enforcement point is typically a Kubernetes admission controller: before a pod is scheduled, it checks that the image carries a valid signature, an SBOM, and an SLSA level meeting policy — and denies admission if any check fails, turning "we scanned it once" into "we verify it every time it runs."

Why isn't a clean build-time scan enough to trust a running workload?

A clean build-time scan isn't enough because the environment a container runs in — and the code paths it actually executes — are only knowable at runtime, and a build-time SBOM can't tell you which of its listed dependencies a given workload ever loads into memory. This is also where supply-chain compromises that evade build scanning eventually surface: Log4Shell was present in artifacts long before most organizations knew to look for it, and the only way many teams found affected services in December 2021 was by watching for the JNDI lookup pattern in live outbound traffic, not by re-scanning source. Runtime observability closes that blind spot. eBPF has become the dominant mechanism for this on Linux because it lets a collector watch process execution, file access, and network connections directly from the kernel without loading a custom kernel module, keeping overhead low enough to run on every node in a fleet continuously rather than as a periodic scan.

How should runtime detections be prioritized once they start firing?

Runtime detections should be prioritized against a standard adversary-technique taxonomy, not treated as an undifferentiated alert stream, because a fleet running eBPF-based sensors at scale can generate far more events than a team can triage by hand. MITRE ATT&CK is the standard framework for this: a reverse shell spawned inside a container maps to T1059, a container-escape attempt via namespace breakout or a privileged mount maps to T1611, and sustained high CPU paired with known mining-pool connections maps to T1496 (resource hijacking). Mapping every rule to an ATT&CK technique ID gives a security team a shared vocabulary with incident response and lets them build policy in terms of adversary behavior — "auto-alert on any T1611 container-escape indicator" — instead of raw log-pattern matches that break every time an attacker changes a command string.

What changes when build, deploy, and runtime evidence are actually connected?

What changes is prioritization: a finding that is confirmed at more than one stage of the lifecycle outranks one that only ever showed up once. A vulnerable function flagged by a build-time scan is a theoretical risk; the same function shown to actually load and execute in a running workload — something only a runtime SCA/reachability signal can confirm — is a materially different risk and deserves to jump the queue. The same logic applies to a workload that is simultaneously running, network-reachable, exposed to an untrusted boundary, and over-privileged: each of those four signals comes from a different stage of the lifecycle (deploy-time RBAC, network policy, runtime process state), and a finding that lights up all four is far more urgent than one that lights up a single dimension. Without a shared correlation key across build provenance, deploy-time attestation, and runtime detection, every stage re-litigates severity from scratch.

How Safeguard connects the three stages

Safeguard produces SLSA provenance, CycloneDX and SPDX SBOMs, and vulnerability-scan attestations for every artifact a pipeline builds, signed via Sigstore's Fulcio/Rekor by default (or X.509/Notation for regulated environments), and a Kubernetes admission controller verifies signature validity, SBOM presence, and SLSA level against policy before any pod is admitted — so a build-time attestation is the same evidence checked at deploy time, not a separate claim. That evidence keeps working after deployment: the runtime rule engine ships ATT&CK-mapped detection packs for reverse shells, container escapes, crypto-miners, and privilege escalation, and its runtime SCA layer records which libraries and functions actually load per workload, feeding AutoTriage so a statically-reachable CVE gets re-ranked up when confirmed live — reachability only ever re-ranks CVE findings, it never suppresses a malware detection. The rule engine itself is decoupled from how those events reach it: the eBPF-based runtime collector that feeds it live workload data is a Linux follow-on still rolling out, not yet claimed as generally available, so detection coverage today depends on that collector's rollout to a given fleet. The CNAPP layer then correlates the running/reachable/exposed/over-privileged dimensions across all three stages, so a finding your team investigates has already been checked against build provenance, deploy-time signature policy, and live runtime behavior — one lifecycle, not three separate tools comparing notes after the fact.

Never miss an update

Weekly insights on software supply chain security, delivered to your inbox.