Many programs list items in an order that changes between runs. Hash maps and sets are randomised on purpose, and filesystems return directory entries in no promised order. If the build writes those items out, the output differs. The fix is to sort explicitly, using a locale independent order.
Three common sources
Directory listings. Most filesystems do not guarantee that listing a directory gives the same order every time. A build that packs everything in a folder into an archive can therefore produce a different archive on each run. The Reproducible Builds docs show tar descending a directory in filesystem order, and recommend listing files with find, sorting with the C locale, and passing the list to tar.
Wildcards in build files. The docs show a GNU Make file where the wildcard function sorts according to the current locale. Two machines with different locales can then compile and link files in different orders. The fixes are to list inputs explicitly, or to wrap the wildcard in Make's sort function, which does not depend on locale.
Hash based data structures. Perl hashes, Python dictionaries and sets, Rust HashMap and HashSet, and Ruby Hash objects can list keys in a different order on every run. The randomisation limits algorithmic complexity attacks, so it is a feature, not a bug. It becomes a problem when a build step writes those keys into a manifest, a generated source file or a metadata file.
What to change
| Language or tool | Option from the docs |
|---|---|
| Perl | Sort the keys explicitly, or set PERL_HASH_SEED to 0 |
| Python | Set PYTHONHASHSEED to a fixed integer |
| Rust | Replace HashMap with BTreeMap where order reaches output, notably in build.rs |
| Make | Sort wildcard results with the sort function, or list files explicitly |
| tar | Sort file names, using the C locale if using find and sort |
Sorting in the program is more durable than setting a seed. A seed fixes today's behaviour for one interpreter. Explicit sorting says what you want in the code, so the next maintainer sees it.
It happens in compilers too
The Go team's post on reproducible toolchains describes the same problem inside the Go toolchain. Map iteration and goroutines whose work was serialised with locks introduced randomness in the order results were generated. To make builds reproducible, the team found each case and sorted the relevant list before generating output.
It also describes a subtler version. A library may choose among several correct outputs. The standard sort package can place elements that compare equal in any order, and that choice can change between Go versions. If a compiler output depends on that, builds from different bootstrap toolchains stop matching.
The lesson for application teams: sorting by a key that can tie is not enough. Make the sort key unique, for example by name plus path, so there is one right answer.
Find the culprit
- Build twice and compare checksums.
- Run diffoscope on the outputs. Reordered lines in a generated file or reordered archive members point to ordering.
- Search the code that generates the file for loops over maps, sets or directory listings.
- Sort, rebuild, compare.
If the difference only appears when tests or code generation run in parallel, look for races as well. The Reproducible Builds commandments say to do one thing at a time or make sure races do no harm.
Quick check
Is setting PYTHONHASHSEED a good long term fix?
It works for Python, but explicit sorting is clearer and survives a change of interpreter or seed handling.
Does sorting with the sort command always work?
Only if you fix the locale. Use LC_ALL=C so every machine orders text the same way.
Safeguard's SBOM and dependency scanning is generally available, and an SBOM is a good example of an output that should list components in a stable order. The free scan at gold.safeguard.sh/scan is a quick way to try it.
Sources
- Reproducible Builds: Stable order for outputs
- Reproducible Builds: Stable order for inputs
- Reproducible Builds: Commandments
- Go blog: Perfectly Reproducible, Verified Go Toolchains
Last reviewed: 19 October 2026.
