The discovery of the XZ Utils backdoor (CVE-2024-3094) will be studied in security courses for decades. Not because the backdoor was technically brilliant, though it absolutely was, but because of how it was found. The most dangerous supply chain compromise in open source history was detected by a single engineer who noticed his SSH logins were slightly slow.
This is the story of that discovery.
The Setup
Andres Freund is a principal software engineer at Microsoft who works primarily on PostgreSQL. In late March 2024, he was running some micro-benchmarks and automated tests on a Debian sid (unstable) system. During this work, he noticed something odd: SSH connections to his machine were consuming more CPU than he expected, and logins were taking roughly 500 milliseconds longer than they should.
For most people, a half-second delay in SSH login would be imperceptible or easily dismissed. Maybe the network was slow. Maybe the server was under load. Maybe it was just one of those things.
Andres Freund is not most people.
Following the Thread
Instead of shrugging it off, Freund started profiling the SSH daemon to understand where the extra CPU time was going. What he found was strange: sshd was spending significant time in the liblzma library during the authentication phase of an SSH connection.
That should not happen. SSH does not use LZMA compression during authentication. The liblzma library is loaded by sshd on systemd-based distributions because sshd links against libsystemd for notification purposes, and libsystemd links against liblzma for journal decompression. But the library should be essentially dormant during SSH authentication.
Freund dug deeper. He examined the recent changes to the XZ Utils package on his system and noticed it had been updated to version 5.6.1. He then looked at what had changed between 5.6.0 (the previous version he had) and earlier versions.
Unraveling the Obfuscation
What Freund found in the build system was deeply suspicious. The release tarballs contained modifications to the autotools build scripts that were not present in the git repository. Specifically, a file called build-to-host.m4 contained obfuscated shell commands that would extract data from files in the tests/ directory and incorporate them into the build.
He traced the obfuscated code through multiple stages of extraction and decompression. The test files bad-3-corrupt_lzma2.xz and good-large_compressed.lzma were not actually test data at all. They were containers for a precompiled shared object that got linked into liblzma during the build process.
Freund decompiled the resulting binary and found code that hooked the RSA_public_decrypt function, which is used during SSH public key authentication. The hooked function checked incoming data against what appeared to be an attacker-controlled cryptographic key.
The Disclosure
On March 29, 2024, at 09:51 UTC, Freund sent his findings to the oss-security mailing list with the subject line "backdoor in upstream xz/liblzma leading to ssh server compromise." The email was measured, technically precise, and devastating in its implications.
He wrote: "After observing a few odd symptoms around liblzma (part of the xz package) on Debian sid installations over the last weeks (logins with ssh taking a lot of CPU, valgrind errors) I figured out the answer: The upstream xz repository and the xz tarballs have been backdoored."
He included a detailed technical breakdown of the backdoor mechanism, the build system manipulation, and the code paths he had identified. He also noted the key observation that made others take immediate action: the backdoor was only included in release tarballs, not in the git repository, meaning that anyone building from the git source would not reproduce the issue.
The Response
The response was immediate and massive. Within hours:
- Debian pushed an emergency update reverting to XZ Utils 5.4.5
- Red Hat/Fedora issued an urgent advisory for Fedora 40 and Fedora Rawhide
- openSUSE reverted their Tumbleweed builds
- CISA issued an alert recommending immediate downgrade
- GitHub suspended the JiaT75 account and restricted the repository
By March 30, essentially every major Linux distribution had issued advisories and pushed reverted packages. The speed of the response was remarkable, enabled entirely by the quality and clarity of Freund's initial disclosure.
Why Automated Tools Missed It
The natural question is: why did no automated security tool, no CI pipeline, no dependency scanner catch this before a human noticed a performance anomaly?
Several factors conspired to make this backdoor invisible to existing tools:
The payload was not in the source code. Static analysis tools examine source code. The backdoor payload was hidden in binary test files and only assembled during the build process from release tarballs. Anyone running security scans against the git repository would find nothing.
Release tarballs differed from git. This is actually common in autotools-based projects, as the autoreconf process generates additional build files. The divergence between the tarball and git repository was not unusual enough to trigger alerts.
The behavior was conditional. The backdoor only activated on x86-64 Linux systems running glibc with systemd. It checked for debuggers and specific environment variables. It would not trigger in a sandbox, a container-based build, or under instrumentation.
Binary test files are ignored. Dependency scanners and SAST tools typically skip binary files in test directories. The malicious test files looked like legitimate test data for a compression library.
The social engineering was invisible to tools. No scanner can detect that a trusted maintainer is actually an adversary who spent two years building credibility. Code review tools can flag suspicious patterns, but the actual source commits from Jia Tan were largely innocuous. The malicious content was in the release tarball build process, which receives far less scrutiny.
The Role of Luck and Skill
There is an uncomfortable truth here. The discovery was the product of extraordinary technical skill combined with a healthy dose of luck. Andres Freund happened to be:
- Running Debian unstable, which had the compromised version
- Performing the kind of work where a 500ms difference mattered
- Sufficiently skilled to profile
sshdand trace the issue toliblzma - Experienced enough to recognize that
liblzmaactivity during SSH auth was wrong - Thorough enough to reverse-engineer the entire backdoor chain
- Responsible enough to disclose it properly to the security community
If any of those conditions had not been met, the backdoor would likely have made it into the next stable releases of Debian 13, Ubuntu 24.04, Fedora 40, and RHEL 10. It would have been present on millions of servers within months.
What This Means for the Industry
The XZ Utils backdoor discovery is a watershed moment. It demonstrated that the open source supply chain is both more fragile and more resilient than anyone assumed. Fragile because a single compromised maintainer can introduce a devastating backdoor. Resilient because the open nature of the code ultimately allowed one curious engineer to find it.
But we cannot rely on luck. The industry needs systematic approaches to verifying software integrity, monitoring for behavioral anomalies, and ensuring that single points of failure in critical open source projects are addressed.
How Safeguard.sh Helps
Safeguard.sh addresses the systemic gap that the XZ Utils incident exposed. Our platform provides continuous SBOM monitoring that tracks not just known CVEs but also version changes and supply chain anomalies across your dependency tree. When a compromise like CVE-2024-3094 is identified, Safeguard.sh can immediately identify every deployment in your infrastructure that includes the affected library, including transitive dependencies you may not even know you have. Safeguard.sh also supports policy gates that can block deployments containing packages from compromised version ranges, turning reactive incident response into proactive defense.