The XZ Utils Backdoor: Anatomy of a Sophisticated Supply Chain Attack

This case study details the process by which a backdoor was introduced into the widely used XZ Utils software, how it functioned, and its eventual discovery. This incident highlights significant risks in the open-source software supply chain. 

Background on XZ Utils 

• Regarding XZ Utils, it’s a no-cost, versatile tool primarily used for compressing data. This software is recognized for achieving significant data size reduction thanks to its underlying LZMA compression method. The LZMA algorithm itself is noted as an advancement of the Lempel-Ziv approach to data compression. Given its effectiveness, XZ Utils has become a commonly integrated component within numerous Linux distributions and software collections. 

• XZ compression is also used internally by some programs, making the xz-utils package a necessary part of many Linux systems. The package includes the liblzma library, which provides the core compression and decompression functionality. 

Step 1: The Attacker’s Infiltration and Gaining Trust 

• In *2021, using the pseudonym *JiaT75 (Jia Tan)an attacker created an account on GitHub and began contributing to various open-source projects. This activity likely served to build credibility. 

• Jia Tan submitted a patch to the libarchive project, which was merged without much discussion. This patch replaced a safe function (safe_fprintf()) with an unsafe variant (fprintf()), potentially introducing another vulnerability. The security implications of this change are still under investigation. 

• Following this, Jia Tan began pressuring the original maintainer of the xz package, Lasse Collin, to add another maintainer to the project. Several other suspicious accounts with little prior activity, such as misoeater91 and krygorin4545 (whose PGP key was created just before joining the discussion), also pushed for this. 

• By February 6th, 2022, Jia Tan submitted their first legitimate commit to the XZ repository. 

• Over time, Jia Tan became increasingly active in the xz project, eventually becoming the *second most active contributor. By *November 30th, 2022, the bug reporting email for XZ Utils was changed to an alias redirecting emails to both Lasse Collin and Jia Tan. 

• In March 2023, Jia Tan made their first release of XZ Utils, version 5.4.2, after Lasse Collin’s final release (5.4.1) in January 2023. This marked a significant step towards Jia Tan taking over the project. 

• There was a social engineering campaign to pressure Lasse Collin to give Jia Tan maintainer rights. Eventually, Jia Tan succeeded in fully taking over the project on GitHub. 

Step 2: Introduction of the Malicious Code 

• Around June 2023, changes were made to XZ Utils that might have set the stage for the attack, including the introduction of ifunc implementation to crc64_fast.c. The implementation of ifunc is  possibly one of the ways the backdoor operates. 

• In July 2023, Jia Tan opened a Pull Request in Google’s oss-fuzz project to disable ifunc fuzzing for XZ, potentially to prevent the fuzzer from detecting the malicious changes. Jia Tan had also changed the primary email address for XZ in oss-fuzz to their personal Gmail address, possibly to intercept vulnerability reports. 

• On February 15th, 2024, Jia Tan added an ignore rule for build-to-host.m4 in the XZ repository’s .gitignore file. This script, later included in release bundles, contained malicious M4 macros to install the backdoor. 

• On February 23rd, 2024, the obfuscated binary backdoor was added in two test files within the XZ repository: tests/files/bad-3-corrupt_lzma2.xz and tests/files/good-large_compressed.lzma. 

• On *February 24th, 2024, *version 5.6.0 of XZ Utils was released, containing the malicious build-to-host.m4 and the backdoor in the test files. This version was pulled by Debian, Gentoo, and Arch Linux. 

• On *March 9th, 2024, Jia Tan released *version 5.6.1, which included an improved version of the backdoor binaries. This version was also pulled by Fedora, Gentoo, Arch Linux, openSUSE, and Alpine. Debian pulled version 5.6.1 on March 26th. 

Step 3: The Backdoor Mechanism 

• The backdoor is not present in the source code itself. Instead, it is hidden in separate “test” files and is reassembled and inserted into the liblzma library during the compilation process if specific conditions are met. The malicious code is only included in the packaged tarball, not in the individual files of the repository, aiding in its stealth. 

• The backdoor targets the OpenSSH server (SSHD). 

• The malicious payload gets injected into the sshd process because liblzma is a dependency of certain builds of OpenSSH, particularly in systems configured to work with system notifications from systemd (notably in Debian Linux). sshd has an indirect dependency on liblzma through systemd. 

• The backdoor hooks the RSA_public_decrypt function within the OpenSSL library (used by OpenSSH for cryptographic functions). 

• The malicious hook inspects the RSA public modules (“N value ”) supplied to the RSA_public_decrypt function, which is under the control of the connecting SSH client(i.e., the attacker). 

• By examining the first 16 bytes of the “N” value, the backdoor derives a “Command Number” that determines its operation, including SSH authentication bypass and remote shell command execution. 

• The backdoor decrypts the last 240 bytes of the “N” value using a hardcoded ChaCha20 symmetric stream cipher key. This key could be used to decrypt network captures of attack attempts. 

• The decrypted data contains a signature that is checked for validity using the Ed448 asymmetric elliptic curve signing algorithm and a hardcoded Ed448 public key. Only attackers with the corresponding private key can generate valid payloads. The signature is also bound to the host’s public key, preventing reuse on different hosts. 

• If the signature is valid, the backdoor uses the bytes following the signature as command-specific payload data, such as a shell command to execute. 

• If the payload or signature is invalid, the original RSA_public_decrypt function is resumed normally, making detection difficult. 

• When activated by a specially crafted SSH connection with the correct private key, the backdoor could allow the attacker to bypass SSH authentication entirely or execute arbitrary code remotely on the affected system before the authentication step. 

Step 4: Discovery of the Backdoor 

• On March 29, 2024, the backdoor was discovered by Andres Freund.  

• Freund noticed unusual behavior associated with sshd, specifically that it was consuming a surprisingly large amount of CPU during the login process. 

•  While using the Valgrind tool for profiling and memory debugging, he encountered multiple errors, which led him to conduct a deeper investigation. Valgrind is a tool used by system programmers for memory debugging and leak detection. 

• Freund’s keen observation and technical skills led to the discovery and public reporting of the backdoor on the oss-security mailing list. 

Step 5: Impact and Significance 

• The XZ backdoor was a significant supply chain attack that could have had severe consequences due to the widespread use of XZ Utils in Linux distributions. 

• If undetected, the backdoor could have allowed attackers to gain unauthorized remote access and execute code on millions of Linux systems. This could have been the most serious software supply chain compromise since the SolarWinds Orion hack. 

• The incident raises serious concerns about the trust and security of open-source software and highlights the risks associated with relying on third-party dependencies. 

• The sophistication and stealth of the attack, along with the multi-year effort to introduce it, suggest the involvement of an advanced threat actor, possibly even a nation-state. 

• The incident underscores the importance of having “enough eyes” on open-source code (Linus’s Law) and how community vigilance can uncover even highly sophisticated threats. 

Step 6: Mitigation and Detection 

• Upon discovery, affected Linux distributions and users were advised to downgrade to earlier, unaffected versions of xz-utils (prior to 5.6.0). 

• It was also recommended to reboot machines or restart the OpenSSH server after downgrading to remove the patched code from memory. 

• A potential workaround, if downgrading was not immediately possible, was to add a specific string (yolAbejyiejuvnup=Evjtgvsh5okmkAvj) to /etc/environment, which would act as a “kill switch” for the backdoor after restarting SSH and Systemd. 

• The JFrog Research team released an open-source detector tool to check if a machine is vulnerable and currently affected by the malicious XZ version. This tool looks for specific characteristics of the malicious code, including encrypted strings. 

• JFrog Xray can also be used to identify vulnerable occurrences of xz_utils in codebases and artifacts. 

This incident underscores the persistent nature of evolving threats to the software supply chain and the vital role of community vigilance, thorough code review, and robust security practices in safeguarding the integrity and resilience of open-source software.  The dedication and sophistication displayed by the attacker highlight the need for continuous improvement in security measures and a cautious approach to trust in the open-source ecosystem.