You review every pull request. You run static analysis on every commit. You enforce branch protection rules. Then your CI/CD pipeline takes the reviewed code, builds it, and produces an artifact that you deploy to production. Do you verify that the deployed artifact was actually built from the reviewed code?
Most organizations do not. The build pipeline is a black box: source code goes in, an artifact comes out, and everyone trusts that the output corresponds to the input. This trust gap is exactly what supply chain attackers exploit. The SolarWinds attack injected code during the build process. The Codecov attack modified a build artifact after it was produced. Both would have been detected by artifact integrity verification.
The Trust Gap
The software delivery chain has multiple points where the artifact can diverge from the reviewed source:
- Build environment compromise. The build server itself is compromised and injects code during compilation or bundling.
- Dependency substitution. A dependency is replaced with a malicious version between code review and build execution.
- Build cache poisoning. Cached build artifacts from a previous compromised build are reused.
- Post-build tampering. The artifact is modified after the build but before deployment (in an artifact registry, during transfer, or in storage).
- Deployment substitution. The deployment system pulls a different artifact than the one that was built and approved.
Integrity verification addresses each of these by creating a verifiable chain from source to deployment.
Cryptographic Signing
What to Sign
At minimum, sign your production build artifacts:
- Container images
- Binary executables
- Package archives (npm packages, Python wheels, Java JARs)
- Deployment manifests
- SBOMs
The signature binds a specific artifact (identified by its cryptographic hash) to an identity (the build system that produced it) and a timestamp.
How to Sign
Sigstore/Cosign is the emerging standard for artifact signing in cloud-native environments. Cosign signs container images and other artifacts using ephemeral keys tied to an OIDC identity. The signature is stored alongside the artifact in the registry.
The workflow:
- Build the artifact in CI/CD
- Cosign signs the artifact using the CI/CD platform's OIDC token (no long-lived keys required)
- The signature is uploaded to a transparency log (Rekor) for public verifiability
- Before deployment, the signature is verified against the expected identity and transparency log
GPG signing is the traditional approach. A GPG key pair is generated, the private key is stored securely, and the build system signs artifacts with the private key. Consumers verify with the public key.
Notation (from the Notary Project) provides a signing framework for OCI artifacts (container images) with pluggable signature formats and trust policies.
Key Management
Signing is only as secure as the key management:
- Never store signing keys in the CI/CD configuration. Use a hardware security module (HSM), a cloud KMS, or Sigstore's keyless signing.
- Rotate keys on a schedule and when personnel changes occur.
- Use separate keys for different environments (development, staging, production).
- Monitor key usage for anomalous signing events (signing outside build pipelines, signing at unusual times).
Provenance Attestation
Signing tells you who signed the artifact. Provenance tells you how the artifact was built. A provenance attestation includes:
- The source repository and commit that was built
- The build platform and configuration
- The build steps that were executed
- The dependencies that were resolved
- The environment variables that were set (excluding secrets)
- Timestamps for each build step
SLSA Provenance
The SLSA (Supply-chain Levels for Software Artifacts) framework defines provenance requirements at four levels:
SLSA Level 1: Provenance exists. The build process generates a provenance document describing how the artifact was produced.
SLSA Level 2: Hosted build. The build runs on a hosted service (not a developer's laptop) and the provenance is signed by the build service.
SLSA Level 3: Hardened builds. The build service provides additional guarantees: builds are isolated, the provenance is non-falsifiable, and the build cannot be influenced by the user beyond the initial configuration.
SLSA Level 4: Full verification. The build is reproducible: given the same source and build configuration, anyone can rebuild the artifact and get the same output.
Most organizations should target SLSA Level 2 as a practical starting point and work toward Level 3.
Generating Provenance
GitHub Actions can generate SLSA Level 3 provenance using the slsa-framework/slsa-github-generator actions. The provenance is generated by a reusable workflow that runs in an isolated environment, making it resistant to tampering by the calling workflow.
Google Cloud Build generates provenance attestations automatically for builds that meet certain criteria.
For other platforms, use the in-toto framework to generate provenance attestations in a standard format that can be verified independently.
Reproducible Builds
The ultimate form of integrity verification: given the same source and build configuration, anyone can rebuild the artifact and verify that it matches the distributed artifact bit-for-bit.
Why Reproducibility Is Hard
Builds incorporate non-deterministic elements:
- Timestamps embedded in artifacts
- File ordering that varies by file system
- Parallelism that affects ordering of operations
- Locale and timezone differences between build environments
- Compiler/interpreter versions that produce different output
Making Builds Reproducible
- Pin all build tools to exact versions (compiler, bundler, runtime)
- Use fixed timestamps or strip timestamps from artifacts
- Sort file lists explicitly rather than relying on file system ordering
- Use deterministic compilation flags
- Document the complete build environment specification
- Use containerized builds with a pinned base image
Verification Workflow
- Obtain the distributed artifact and its provenance attestation
- Extract the source repository, commit, and build configuration from the provenance
- Reproduce the build environment from the provenance specification
- Build the artifact from source
- Compare the hash of the rebuilt artifact against the hash of the distributed artifact
- If they match, the artifact is verified
Deployment Verification
Signing and provenance protect the artifact up to the registry. Deployment verification ensures the correct artifact is actually deployed:
- Image digest pinning. Deploy container images by digest (sha256:abc123) rather than tag. Tags are mutable; digests are not.
- Admission controllers. Use Kubernetes admission controllers (e.g., Kyverno, OPA Gatekeeper, Sigstore Policy Controller) to verify signatures and provenance before allowing deployments.
- Post-deployment verification. After deployment, query the running instance for the artifact hash and verify it matches the expected value.
How Safeguard.sh Helps
Safeguard.sh integrates with your build pipeline to verify artifact integrity at every stage: source provenance, build artifact signing, SBOM accuracy, and deployment verification. Policy gates can require valid signatures and provenance attestations before artifacts are promoted to production, ensuring that the entire chain from reviewed source code to deployed artifact is verifiable. When combined with Safeguard's vulnerability scanning and dependency tracking, integrity verification closes the trust gap between code review and production deployment.