Container Security

BuildKit Cache Security Considerations for Container Builds

BuildKit's caching is what makes container builds fast. It is also a potential vector for cache poisoning attacks if not properly secured.

Yukti Singhal
Security Researcher
5 min read

BuildKit is the modern build engine behind Docker builds, replacing the legacy builder with a more efficient, parallel, and cache-aware system. Its caching capabilities, including layer caching, mount caches, and remote cache backends, make builds dramatically faster. A build that takes 10 minutes without cache might finish in 30 seconds with a warm cache.

But caching introduces security considerations that most teams overlook. The cache is a trusted source of build artifacts. If an attacker can poison the cache, they can inject malicious content into your container images without modifying your Dockerfile or source code.

How BuildKit Caching Works

Layer Cache

BuildKit caches each layer of your Dockerfile. If a layer's inputs have not changed (the Dockerfile instruction, the files it references, and the parent layer), BuildKit reuses the cached output instead of re-executing the instruction.

This is why the standard practice of copying dependency manifests before source code works: the dependency installation layer is cached unless the manifest changes.

Mount Caches

BuildKit supports cache mounts that persist across builds. A --mount=type=cache directive creates a persistent directory that survives between builds:

RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt

The pip cache persists between builds, so packages that were downloaded in a previous build do not need to be downloaded again.

Remote Cache

BuildKit can push and pull cache layers to and from a remote registry. This enables sharing cache across CI/CD runners, across team members, and across environments:

docker buildx build \
  --cache-from type=registry,ref=myregistry.com/cache:latest \
  --cache-to type=registry,ref=myregistry.com/cache:latest \
  .

Cache Poisoning Risks

Shared Cache Between Projects

In shared CI/CD environments, multiple projects may share a BuildKit instance and its cache. If Project A can influence the cache, it can potentially inject malicious layers that Project B uses.

This is most dangerous with mount caches. If two projects use the same cache mount path for their package manager cache, one project can place a malicious package in the cache that the other project installs.

Untrusted Cache Sources

Remote cache from a registry is only as trustworthy as the registry and the builds that populated it. If an attacker can push to your cache registry, they can inject poisoned layers.

When using --cache-from, the cache layers are pulled from the registry and used directly, bypassing the build instructions in your Dockerfile. A poisoned cache layer replaces the output of a Dockerfile instruction with whatever the attacker wants.

Cache Persistence After Compromise

If a build is compromised and produces a malicious layer, that layer persists in the cache. Subsequent builds reuse the poisoned layer until the cache is invalidated. The compromise survives even after the original attack vector is patched.

Dependency Confusion Through Cache

Mount caches for package managers (pip, npm, Maven) store downloaded packages. If an attacker can place a malicious package in the mount cache with a higher version number than the legitimate package, the package manager may install the malicious version.

Securing BuildKit Caches

Isolate Caches Per Project

Do not share BuildKit instances or caches across security boundaries. Each project or trust level should have its own BuildKit instance with its own cache.

In Kubernetes-based CI/CD, use separate BuildKit deployments for different trust levels rather than a shared BuildKit service.

Secure Remote Cache Registries

Treat your cache registry with the same security as your image registry. Restrict push access to authorized build pipelines. Use separate registries or repositories for cache and final images. Monitor for unauthorized cache pushes.

Invalidate Cache After Security Incidents

If a build pipeline or dependency source is compromised, invalidate all caches that the compromised pipeline populated. Do not trust that cleaning up the source of compromise is sufficient. The poisoned cache will continue to serve malicious content.

Use Checksums in Dockerfiles

Where possible, verify checksums of downloaded artifacts within your Dockerfile. Even if the cache serves a previously downloaded file, the checksum verification catches modifications:

RUN --mount=type=cache,target=/tmp/downloads \
    curl -o /tmp/downloads/tool.tar.gz https://example.com/tool.tar.gz \
    && echo "expected_sha256  /tmp/downloads/tool.tar.gz" | sha256sum -c - \
    && tar xzf /tmp/downloads/tool.tar.gz -C /usr/local/bin/

Regular Cache Purges

Periodically purge build caches to limit the persistence of any poisoned content. This trades build speed for security assurance. For high-security builds, consider running without cache entirely.

Separate Build and Production

Build your images on isolated infrastructure that does not share caches with development environments. Production build pipelines should not use caches populated by development or testing builds.

BuildKit Secrets and Cache

BuildKit secrets (--mount=type=secret) are designed to not be cached. However, the output of commands that use secrets is cached. If a secret-using command produces output that embeds the secret value, that value ends up in the cache layer.

Verify that your secret-using commands do not leak secrets into their output. The secret itself is not cached, but the layer that was built using the secret is.

Monitoring

Track Cache Hit Rates

Monitor your cache hit rates. A sudden drop in cache hits might indicate cache invalidation from an unexpected source. A sudden increase in cache hits for a project that was previously cache-cold might indicate cache injection.

Audit Cache Contents

Periodically inspect the contents of your mount caches. Look for unexpected packages, modified files, or files that do not match expected checksums.

Log Cache Operations

Enable BuildKit logging for cache operations. Track which builds push to and pull from cache, and alert on unexpected cache modifications.

How Safeguard.sh Helps

Safeguard.sh scans your container images regardless of how they were built, catching supply chain issues that might enter through cache poisoning or any other vector. It generates SBOMs for your final images, identifies vulnerable components, and provides the verification layer that ensures your container contents match expectations. When a cached dependency has a vulnerability, Safeguard.sh detects it in the final image even if the cache masked the download.

Never miss an update

Weekly insights on software supply chain security, delivered to your inbox.