Reproducible Builds Debian: The Long View

The Reproducible Builds project started as an idea on the Debian mailing list in August 2013 and has quietly become one of the most consequential pieces of software supply chain work in the Linux ecosystem. By the end of 2024, Debian is over a decade into the effort. The percentage of packages in the unstable archive that build reproducibly sits above 97% for amd64, and the tools developed along the way -- diffoscope, strip-nondeterminism, buildinfo files -- have spread to many other distributions.

I have spent enough years in supply chain work to treat Reproducible Builds as table stakes for any argument about binary provenance. This post is the long view: where the project started, what they learned, what still doesn't reproduce, and why someone building production software in 2024 should care.

The Problem Reproducible Builds Solves

Ken Thompson's 1984 Turing Award lecture, "Reflections on Trusting Trust," posed the core question: if you didn't write the compiler, how do you know what it is doing? A malicious compiler can inject back doors into any program it builds, and even into a fresh build of itself, in a way that leaves no trace in the source code.

Reproducible Builds is a partial answer. If a build is reproducible -- meaning multiple independent parties with the same source and build environment produce byte-identical binaries -- then a bad actor can no longer secretly substitute a different binary for an expected source. The build output becomes verifiable, not just trusted.

The practical version of this in 2024: when Debian ships libssl3_3.1.4-1_amd64.deb, any interested party can rebuild from libssl3_3.1.4-1.dsc and check that the output hash matches what Debian publishes. If it does, the binary is provably the output of the source. If it doesn't, something is wrong -- a compiler back door, a tampered build farm, or, more commonly, a source of non-determinism that hasn't been fixed yet.

The Slow Grind of Non-Determinism

Non-determinism in builds comes from everywhere. The first systematic Debian survey in 2015 found seven major categories:

1. Timestamps in output files. The most common source. Tarballs, archives, and generated files embed the current time.
2. Build paths. Debug sections encode the absolute path of source files, which differs between build hosts.
3. Locale dependencies. Tools that sort differently under different LC_COLLATE settings produce different output orderings.
4. Parallel build races. Ordering of output within archives depends on which job finishes first.
5. Umask dependencies. File permissions in archives depend on the builder's umask.
6. Toolchain version drift. A new GCC release emits different code for the same source.
7. Random seeds. Some tools (certain Go versions, some Perl codegen) use /dev/urandom for module IDs.

Each of these required a targeted fix, often upstream in the affected tool. Timestamps went away when SOURCE_DATE_EPOCH became a convention (documented in the Reproducible Builds specification and adopted by gcc, clang, go, cargo, and thousands of other tools). Build paths were addressed by -ffile-prefix-map in GCC and -trimpath in Go. Locale issues get forced to LC_ALL=C.UTF-8 in the build farm. The list goes on.

The key insight was that there is no single patch that makes everything reproducible. Reproducibility is achieved one bug at a time, across thousands of upstream tools. The Debian project has filed thousands of bugs, maintained patches, and pushed changes upstream for over a decade. It is grindwork and institutional patience.

diffoscope: The Debugger of Reproducibility

diffoscope is the tool that makes this work possible. Built by Debian developers starting in 2014, diffoscope takes two files and recursively compares them, descending into archives, decompiling binaries, diffing metadata, and producing a human-readable report.

A typical diffoscope run:

diffoscope /var/cache/pbuilder/result/foo_1.0_amd64.deb \
           /var/cache/pbuilder/result/foo_1.0_amd64.deb.rebuild \
           --html diff.html

The output shows that lib/foo.so has a different build-id, that share/doc/foo/changelog.gz has a different gzip timestamp, that the data.tar.xz inside the .deb has different mtimes. Each difference is a specific bug to fix.

diffoscope has become the standard tool for reproducibility work across the Linux ecosystem. openSUSE, NixOS, Arch, Guix, and individual projects like F-Droid all use it. I have used it for debugging reproducibility issues in everything from Go binaries to Android APKs. It is one of those tools that becomes indispensable the first time you use it.

buildinfo Files: The Provenance Attachment

Debian's .buildinfo file format, standardized in 2016, captures the exact build environment used to produce a binary package. A buildinfo file lists:

• The source package version and hash
• The binary package hashes produced
• Every Build-Depends package and its exact version
• The build architecture
• The build date (as an offset from the source)
• Environment variables that affected the build

Here is a snippet from a real buildinfo file:

Format: 1.0
Source: openssh
Binary: openssh-client openssh-server
Architecture: amd64
Version: 1:9.6p1-4
Build-Date: Tue, 26 Mar 2024 14:32:19 +0000
Build-Path: /build/openssh-9.6p1
Installed-Build-Depends:
 debhelper (= 13.15.3),
 gcc (= 4:13.2.0-7),
 libc6-dev (= 2.37-12),
 libssl-dev (= 3.1.4-2)

The buildinfo file is the Debian-native version of a SLSA provenance attestation, and it predates SLSA by years. The Reproducible Builds team's mid-2020s work has been about connecting buildinfo to the broader attestation ecosystem -- specifically, making buildinfo files readable as CycloneDX or SPDX components for downstream SBOM tooling.

What Still Isn't Reproducible

As of late 2024, the gaps are:

• Packages using non-deterministic compilers. A few languages (Haskell, with its profiling build mode; some Ada compilers) still produce non-reproducible output in certain configurations.
• Packages with randomized address space layout at link time. Some security-hardening linker options introduce randomness that breaks bit-identity. The project has been working with toolchain developers to allow deterministic hardening.
• Hardware-dependent microcode and firmware. Some packages embed CPU-specific microcode that differs between build hosts. These are typically ignored for reproducibility purposes.
• Cross-architecture packages. arm64, armhf, and ppc64el have lower reproducibility rates than amd64 because the build farms see less traffic and bugs take longer to find.

For the vast majority of Debian packages, reproducibility is solved. For the remaining percent, the project continues to grind upstream fixes.

Why It Matters in 2024

The XZ Utils back door, discovered in March 2024, changed the supply chain conversation. The attacker (a maintainer named "Jia Tan") had spent years gaining trust and then inserted malicious code into the xz tarball -- but not into the git repository. The tarball and the git source diverged, and the distribution build consumed the tarball.

Reproducible Builds would not have prevented the insertion. But reproducibility combined with source auditing -- and specifically, the discipline of building from git rather than from release tarballs -- would have made the divergence detectable. The attack worked because the build artifact was trusted beyond the source. Reproducible Builds tightens that trust to the source level.

This is why I view Reproducible Builds as foundational, not optional. Every other supply chain technology -- SBOMs, SLSA, Sigstore -- works better when the underlying builds are reproducible. An SBOM for a non-reproducible build is a snapshot of a process, not a description of an artifact. A SLSA attestation for a non-reproducible build tells you something was built, not that you could rebuild it.

Adopting the Lessons

If you are building software in 2024, the Reproducible Builds work offers a well-tested recipe:

1. Set SOURCE_DATE_EPOCH to a fixed value (the source commit timestamp is conventional).
2. Use -ffile-prefix-map (GCC/Clang), -trimpath (Go), and equivalent flags in your toolchain.
3. Force LC_ALL=C.UTF-8 and TZ=UTC in the build environment.
4. Fix or disable any tool that embeds hostnames, usernames, or PIDs in output.
5. Run your build twice on different hosts and diff the output with diffoscope.

The first four are one-time fixes. The fifth is an ongoing discipline that catches regressions. Teams that do this report catching real supply chain bugs -- misconfigured build farms, tampered binary caches, accidental non-deterministic changes in dependencies.

How Safeguard Helps

Safeguard ingests Debian buildinfo files and the equivalent provenance documents from other distributions to correlate deployed binaries with their source provenance, including the full set of Build-Depends for every artifact. Policy gates can require reproducibility verification for any binary entering production -- including running diffoscope-style comparison against expected outputs and flagging drift. For teams building their own software, Safeguard links SLSA attestations to reproducibility results so an artifact's trust chain includes both "who built it" and "could we rebuild it." That extends Debian's decade of reproducibility work into commercial pipelines, giving security teams the same level of artifact-to-source assurance that the Debian project achieves for its own archive.