Open Source Security

dotnet restore Reproducibility Concerns

dotnet restore is supposed to be deterministic. In practice it is deterministic in ways that matter less and non-deterministic in ways that matter more.

Nayan Dey
Senior Security Engineer
7 min read

The pitch for dotnet restore is that it is reproducible. You have a manifest, you have a set of feeds, you run the command, you get the same graph. This is true in a technical sense and largely true in practice for projects that do everything the modern way. It also misses several failure modes where restores across two machines or two moments in time quietly produce different results, and some of those failure modes have supply chain consequences that matter.

I did a deep dive on this for a customer whose build outputs were not bit-for-bit reproducible despite the team's belief that they should be. What I found is worth writing down because most of the stories on this topic conflate reproducibility of the build (MSBuild output) with reproducibility of the restore (package graph), and the second is where the subtle failures live.

What restore is supposed to do

Given a project with PackageReference entries, dotnet restore resolves the full transitive graph, downloads packages to the local cache, writes a project.assets.json file describing the resolved graph, and creates or updates the lock file if one is configured. If the inputs are identical (the project files, the nuget.config, the feed contents, the local cache state, the NuGet client version) the outputs are identical.

The reproducibility problem is that several of those inputs are less stable than they appear.

Floating version ranges

PackageReference supports version ranges like [6.0.0, 7.0.0) and floating versions like 6.0.*. When a project uses these, the resolver picks the highest version in the range that satisfies all constraints. If a new package version is published to the feed between two restores, the second restore picks a different version. The project.assets.json is different, the downloaded packages are different, and the build can be different.

This sounds like a bug people would avoid, but it is extremely common in practice. Microsoft.Extensions.* packages are frequently referenced with floating versions in .NET project templates. Some third-party packages include version range recommendations in their install documentation. The average .NET repository I audit has at least two floating references somewhere in its project graph.

The fix is to pin every direct reference to an exact version and to use a lock file for transitive pinning. NuGet's packages.lock.json feature, enabled via <RestorePackagesWithLockFile>true</RestorePackagesWithLockFile>, captures the resolved graph and can be enforced on subsequent restores with --locked-mode. Shipping with locked mode in CI is the canonical way to enforce restore reproducibility.

Feed content mutations

The NuGet protocol does not promise that a package version is immutable. nuget.org effectively treats packages as immutable after publish, with the delist operation being a soft deletion rather than a removal. Private feeds vary. Some enterprise feeds allow overwriting a version with new content, either by configuration or by administrative action. This breaks the assumption that "Serilog 3.1.1" means the same bytes to every consumer.

The fix is to enable strict hash verification. NuGet 6.5 improved the support for recording package hashes in the lock file, and --locked-mode will refuse to restore if a cached package does not match the recorded hash. This catches the feed-mutation case at restore time. The .NET 8 SDK made this easier by enabling hash verification by default when a lock file exists.

The local cache ambiguity

NuGet has a global packages folder, by default at ~/.nuget/packages on Unix or %USERPROFILE%\.nuget\packages on Windows, where all restored packages are cached. Two restores on the same machine can produce different results if the cache state is different between them. A common case is where a first restore populates the cache with version X of a package, and a later change to the manifest causes a restore that would resolve a different version Y, but a partial manifest state causes the resolver to use the already-cached X.

Most of these cases are transient and resolved by re-running the restore, but they can fool diff-based CI reproducibility checks into passing when the underlying graph is not truly stable. The --force flag on restore is the nuclear option; more surgical is to isolate the global packages folder per build using the NUGET_PACKAGES environment variable or the <RestorePackagesPath> property.

Client version differences

Different NuGet client versions have different resolver behaviours. The 3.x resolver, the 4.x resolver, the 6.x resolver all have slight differences in how they handle edge cases in version ranges, how they interact with central package management, and how they merge transitive graphs. A project restored with dotnet SDK 6.0.400 and the same project restored with SDK 8.0.100 can produce different project.assets.json outputs.

This matters for reproducibility because build agents upgrade on their own cadence and developer workstations often run different SDK versions than CI. The global.json file is the .NET mechanism for pinning the SDK version, and projects that care about reproducibility should use it. Including global.json in the repository and enforcing it in CI removes this source of variation.

The package signing verification interaction

Reproducibility interacts with signature verification in a subtle way. If your nuget.config requires signatures and a cached package was fetched before signing was enforced on the feed, a restore with --locked-mode might succeed while a restore without the cache would fail, because the signature verification happens at fetch time and the cached package was fetched under different rules. The failure mode is asymmetric and hard to diagnose because the symptom is "works on my machine, fails in CI" or vice versa.

The discipline is to treat the local cache as build-agent-private, not to share caches across agents with different signature policies, and to periodically clear the cache on CI agents to force re-fetch under current policy. We do this weekly on our production agents; it catches signature-policy drift that otherwise sits silent.

CVE-2023-36874 and restore timing

A specific incident worth mentioning. In July 2023 Microsoft disclosed CVE-2023-36874, a Windows Error Reporting elevation-of-privilege bug that was being exploited in the wild. The NuGet client running on vulnerable Windows hosts was one of several processes attackers used to pivot. The remediation involved updating Windows, but it also drew attention to the local cache as a place where untrusted content lives. A restore that pulls in a package, writes its files to the cache, and then has its output consumed by other processes is a small but real attack surface.

For reproducibility purposes the relevant observation is that the cache is shared between restores, and anything that happens to the cache between restores is a reproducibility hazard. A malicious process tampering with cached packages would break restore reproducibility and could go undetected if the lock file's hash check is not enabled.

The checklist that actually works

For a .NET project to be genuinely reproducible on restore I want all of these: pinned exact versions on all direct references, a packages.lock.json with RestorePackagesWithLockFile set, CI builds using --locked-mode, a global.json pinning the SDK version, a nuget.config committed with the repository, signature requirement on the feed, and a build-agent-local NUGET_PACKAGES folder not shared between unrelated builds.

Most .NET repositories I audit have two or three of these. Getting all seven turned on is a manageable afternoon of work and tightens the reproducibility story substantially.

How Safeguard Helps

Safeguard tracks whether each .NET repository has lock files, pinned versions, an SDK-pinning global.json, and enforced signature verification, and it reports the aggregate reproducibility posture as a measurable tenant-level score. When a package in a locked restore changes hash on subsequent CI runs the platform flags it as a tamper indicator, which catches feed mutations before they produce unexplained build differences. The platform also correlates the NuGet client and SDK versions used across builds so version drift is visible without scraping CI logs. For customers migrating toward reproducible builds the remediation plan is produced against the checklist above and tracked until each item is green.

Never miss an update

Weekly insights on software supply chain security, delivered to your inbox.