Introduction
The recent XZ Utils supply chain security incident has sent shockwaves through the industry. XZ is a general purpose data compression format present in nearly every Linux distribution, both community projects and commercial product distributions.
As per the CVE-2024-3094, the build process extracts a disguised test file existing in the source code, which is then used to maliciously modify specific functions in the liblzma code to inject the backdoor.
We analyzed the XZ Utils build process using StepSecurity Harden-Runner and observed the injection of the backdoor.
Check out the full analysis of XZ Utils build process in this video here or read the blog post:
The Incident and the Attacker's Strategy
The incident involved the following malicious modifications before the build process:
- Modification of build-to-host.m4 file: This file was altered in the release tarball for versions 5.6.0 and 5.6.1.
- Addition of compressed files: Two malicious files were compressed and added to the tests/files folder in the source code repository.
The attacker deliberately avoided embedding an object file directly in the source code or modifying the build instructions to link it, as such actions would raise suspicion. Instead, they chose a subtler approach by adding test files and modifying the build-to-host.m4 file in the release tarball (which was not in the source code). This strategy ensured that the malicious modifications would occur during the build process, which is typically not closely monitored. As a result, this malicious modification would not be detected by static analysis tools. The attacker counted on this lack of monitoring to evade detection.
Harden Runner: Securing the Build Process
StepSecurity Harden-Runner provides network egress control and CI/CD infrastructure security for GitHub-hosted and self-hosted runner environments. It has been leveraged by Microsoft, Google, CISA, DataDog, Intel, and hundreds of other organizations to enhance their GitHub Actions security.
Key features of Harden Runner include:
- Real-time Monitoring: Continuously observes file write operations during the build process for unusual activities.
- Anomaly Detection: Identifies deviations from the expected build behavior, flagging potential security threats.
- Detailed Insights: Provides comprehensive reports and insights into the build process, helping to pinpoint the exact nature and source of the issue.
By integrating Harden Runner into your build system, you can enhance your security posture and ensure that any tampering is promptly detected and mitigated.
Analyzing the Incident with Harden Runner
GitHub Repository and Workflow
The code being built is hosted on this GitHub repository. The code was originally sourced from Debian's repository, specifically the debian/5.6.0-0.2 tag.
The GitHub Actions workflow used to run the steps is configured to perform the following:
- Checkout the code from the repository
- Run the configure script to generate the Makefile
- Execute the make step to build the XZ Utils binary
The workflow integrates Harden Runner to monitor and analyze the build process. Here are the Harden Runner insights from the workflow execution:
The “File Write Events” tab shows each file that was written to during the build process and what process wrote to it, along with the process arguments.
The XZ Utils Build Process
The build process for XZ Utils is divided into two main parts:
- Configure Step: This step involves running the configure script to generate the Makefile
- Make Step: The Makefile is then used to build the XZ Utils binary
1. Configure Step:
During the configure step, the tampered build-to-host.m4 file caused the Makefile to be modified. Multiple sed (Stream editor) commands were executed to alter the Makefile in place, adding a reference to one of the test files named bad-3-corrupt_lzma2.xz.
Harden Runner Analysis: Using Harden Runner, we observed these changes in real-time. Below are screenshots capturing the suspicious modifications during the configure step.
In the file write events viewer, clicking on “src/liblzma/Makefile” shows the different processes that write to the Makefile. In the build process for the previous non-tampered release tarballs, Makefile is written to by the first mv process only.
On the other hand, in the build for version 5.6.0, you can clearly see multiple sed commands writing to the Makefile. This is done due to the malicious code in the build-to-host.m4 file, which is in the “m4” folder and run during the configure step.
Clicking on any sed commands and walking up the process tree show the process arguments.
Harden-Runner gets this information by monitoring process calls during the build process. For process with the PID: 6645 you can see the that a reference to the file “bad-3-corrupt_lzma2.xz” from the tests/files folder is added to the Makefile. For the process with PID: 6650 you can see that the “xz -d” command is added to the Makefile. This will later cause the “bad-3-corrupt_lzma2.xz” to be unzipped and executed using bash (more on this later).
2. Make Step:
In the make step, the modified Makefile ran, leading to the following suspicious activities:
- Creation of Object File: Another test file was uncompressed to create an object file, which is unusual as object files are typically created using compilers
In the "src/liblzma" folder, if you look for the “liblzma_la-crc64-fast.o” file and click on it, you will notice that it was written by dash (Debian Almquist Shell), which is a POSIX-compliant implementation of bash). Compare that to another object file in the screenshot which has been written to by linux-gnu-as, which is an assembler that is used to assemble the output of the GNU C compiler for use by the linker.
If you click on the dash process, you can move up the process tree and look at the process arguments.
Here you can see that the test file "good-large_compressed.lzma" is uncompressed, there is some manipulation on the file, and it is again uncompressed and then part of it is taken and written as the object file. This is a huge red flag during the build process, as object files should be created by compiling and linking code and not by uncompressing zipped test files. You will see later how this object file is linked to create another object file, which is then used to create the xz utils binary.
To understand what caused this object file to be created by uncompressing a test file, you can move up the process tree.
You can see that the process to uncompress the ”good-large_compressed.lzma” file was started by uncompressing the “bad-3-corrupt_lzma2.xz” file and executing it. If you remember the previous tampering with the Makefile, you will remember seeing these statements inserted into the Makefile.
- Modification and Piping of Source Code File: A source code file was modified and piped as input to a compiler along with the object file. This is irregular because code files are not typically sent to a compiler as piped input.
In the "File write event" viewer if you go to the "src/liblzma/.libs" folder, you will notice that a couple of object files have been overwritten. If you click on the “liblzma_la-crc64_fast.o” file and the “/usr/bin/x86_64-linux-gnu-ld.bfd (PID: 14551)” process, you can see what process wrote the file the second time.
Here you can see the "src/liblzma/check/crc64_fast.c” file is read and modified in memory by process ID 14544. This modified file is not written to disk but the "-x c –" arguments in the process 14547 take the modified source code file from the previous process as an input. This is a huge red flag since compilers should read files from disk and not take them as input arguments. You will also notice in the process 14547 that the object file "liblzma_la-crc64-fast.o" that was written by uncompressing a test file has been linked in this step, and this is how the xz binary was backdoored.
Conclusion
The XZ Utils incident underscores the importance of securing build environments. The tampered build-to-host.m4 file and the addition of compressed files to the tests/files folder resulted in a series of unusual and suspicious activities during the build process. Harden Runner proved to be an invaluable tool in detecting and analyzing these changes.
By integrating Harden Runner into your build system, you can significantly enhance your security posture, ensuring that any tampering or malicious modifications are promptly identified and mitigated.
If you're responsible for managing build environments, consider implementing Harden Runner to safeguard your systems against similar incidents. To get started with Harden-Runner, visit: https://www.stepsecurity.io/starting-free-trial
