StepSecurity Detects CI/CD Supply Chain Attack in Google’s Open-Source Project Flank in Real-Time

Introduction

An independent security researcher recently demonstrated a successful supply chain attack on Flank, an open-source project maintained by Google. A vulnerable GitHub Actions workflow led to this attack. The researcher successfully exploited the vulnerability and gained access to the workflow's GITHUB_TOKEN, which had "write-access" to the repository. This means that the token could be used to overwrite the project releases, potentially leading to a supply chain attack. As Flank had been using StepSecurity Harden-Runner, this attack was detected in real-time, even though the researcher tried to circumvent Harden-Runner's defenses by crafting an innovative exploit. StepSecurity Harden-Runner provides network egress control and CI/CD infrastructure security for GitHub-hosted and self-hosted environments. For executive summary, check out the video below.

Now, let’s understand the vulnerability, how it was exploited, how Harden-Runner detected it in real time, and more.

What was the vulnerability?

One of the workflows in the Flank project on GitHub was set up to run whenever a new comment was made on its associated pull request. When a workflow is run from a forked repository, the GITHUB_TOKEN is usually set to read-only. However, in this case, the workflow was able to bypass this restriction by running when a comment was created and running the untrusted code from the forked repository with elevated GITHUB_TOKEN permissions. This is the vulnerable workflow:

https://github.com/Flank/flank/actions/runs/8074974843/workflow

Here are the vulnerabilities that led to the attack:

Elevated GITHUB_TOKEN permissions

The screenshot below from the build log shows the list of permissions assigned to the job’s GITHUB_TOKEN. The "contents: write" permission can be used to push code to the repository and update the project’s releases.

Check out of untrusted code from pull requests

The screenshot below shows the vulnerable job checking out code from the pull request when running on an issue comment event (line 104).

Execution of untrusted code

As you can see in the screenshot below, the untrusted code is then executed in the Gradle Integration Tests step (line 131).

How was the vulnerability exploited?

The security researcher exploited the vulnerability by creating a pull request from a fork and then creating a comment in the pull request to trigger the workflow with elevated GITHUB_TOKEN permissions. The pull request had code added to a test case to download and execute code from the fork. This malicious code then exfiltrated the GITHUB_TOKEN for the job to a gist in the researcher’s account.

This is the pull request created by the security researcher:

https://github.com/Flank/flank/pull/2481/files

The screenshot below shows the code added in the pull request. You can see that it fetches code from a commit and then runs it using bash (line 17).

The screenshot below shows the code fetched from the commit. This code downloads a Python code snippet, which steals the GITHUB_TOKEN from the Runner.Worker process memory and exfiltrates it to a researcher-controlled destination on GitHub.

https://raw.githubusercontent.com/flank/flank/128b43b61fd7da13ea6829d1fbb4d3f028b6cdad/LICENSE

What could have happened in a real malicious attack?

This would have caused an XZ Utils and SolarWinds style software supply chain attack. By maliciously tampering with the existing software releases, an adversary could have added a backdoor to them. This would have compromised all users of the Google Flank project.

How did StepSecurity Harden-Runner detect this attack in real time?

The Flank maintainers had added Harden-Runner to their workflows, and thus, the Flank project has been using StepSecurity Harden-Runner in the affected workflow since December 2022 (see line 91 in the screenshot below). Harden-Runner was being used in audit mode, and each outbound call for each run of the job has been meticulously logged and monitored since then.

Harden-Runner created a baseline for the job's outbound traffic based on previous outbound calls. When the researcher exploited the vulnerability, an outbound call was made to a new endpoint – raw.githubusercontent.com, which was not in the baseline. This caused a detection to be triggered. Let’s compare the runtime insights generated by Harden-Runner for a non-malicious run with the malicious one.

Runtime Insights for a Non-Malicious Workflow Run

These are the Harden-Runner insights for a run before the attack happened. You can access these insights at https://app.stepsecurity.io/github/Flank/flank/actions/runs/8073347496. Harden-Runner insights are public for open-source projects and require authorization for private repositories.

As you can see, the step that runs the test cases only makes calls using the Java process to the following list of endpoints.

Insights for the Malicious Workflow Run with Credential Exfiltration

These are the insights for the run in which the researcher exfiltrated the GITHUB_TOKEN. You can access these insights at:

https://app.stepsecurity.io/github/Flank/flank/actions/runs/8074974843

In this case, the curl process is making additional outbound calls in the same step, and a call is made to a new endpoint, which has been flagged as anomalous.

This detection shows the power of Harden-Runner and how it successfully detected a sophisticated supply chain attack by an expert researcher!

Here’s what Adnan Khan, the researcher who carried out this attack, had to say about StepSecurity Harden-Runner:

"Ran into StepSecurity Harden-Runner while executing a PoC against a Google OSS repository and it picked up an anomalous curl to raw.githubusercontent.com despite some initial effort I made to blend in. The maintainers had Harden Runner in audit mode, but that telemetry could very well be the difference between a supply chain attack and successful incident response for an organization that actually alerts on it. Hats off to what StepSecurity has built. It works."
‍
Adnan Khan
Independent Security Researcher

Adnan has published a detailed blog post on his research findings here.

What additional StepSecurity features, if used, could have made this attack harder to execute?

Monitoring of HTTPS traffic

The security researcher knew that Harden-Runner was monitoring the vulnerable workflow and tried to exfiltrate the GITHUB_TOKEN using github.com and api.github.com endpoints. Since this job already made calls to these endpoints, and they were part of the baseline, detecting the attack would have been harder. It so happened that the exploit code also made a call to raw.githubusercontent.com, which was not in the baseline and triggered a detection.

Harden-Runner also supports monitoring outbound HTTPS traffic to github.com and api.github.com endpoints as part of the enterprise tier. This monitoring provides additional details, like the method (GET/ POST, etc.) and the path of HTTPS requests made to GitHub APIs. With HTTPS monitoring enabled, this attack would have been detected even if outbound calls were only made to github.com and api.github.com endpoints, as the call to make a Gist (using the path https://api.github.com/gists) would have been flagged as suspicious.

Setting minimum GITHUB_TOKEN permissions

In this case, the job’s GITHUB_TOKEN has all the available permissions, including contents: write, which would have allowed the researcher (or an actual attacker) to overwrite this project's releases, leading to a supply chain attack.

Each GitHub Actions job run has a unique GITHUB_TOKEN, and developers should set the minimum token permissions based on the job's needs.

StepSecurity helps set the minimum token permissions by calculating the required permissions based on the outbound HTTPS calls to the GitHub APIs. This is another reason why enabling HTTPS monitoring would have helped, as it would have provided a recommendation for the minimum GITHUB_TOKEN permissions based on previous calls to the GitHub APIs.

Running jobs without sudo access

The exploit code used by the researcher needed sudo access to exfiltrate secrets from the GitHub-hosted runner. However, the typical runs of this workflow did not need sudo access. Harden-Runner supports an option to disable sudo while running jobs which is supported for exactly such scenarios. Had the disable sudo option been used, the exploit code would have failed since it needed sudo to run.

Setting a network egress block policy for the job

Harden-Runner can detect anomalies in audit mode and can also be configured to run in block mode. The block mode only allows calls to destinations in the baseline and blocks all other calls in real time. Had the job used Harden-Runner in block mode, the exploit code would have failed to exfiltrate the token, as the call to raw.githubusercontent.com would have been blocked.

Conclusion

The security researcher’s demonstration of a supply chain attack on Flank and the detection of the same by StepSecurity Harden-Runner underscores the importance of securing CI/CD pipelines from emerging threats. We applaud the efforts of the maintainers of Flank for staying vigilant and using the StepSecurity platform to secure their workflows. Also, kudos to the security researcher for highlighting the vulnerabilities with ethical testing, raising industry awareness about CI/CD security and securing the open-source ecosystem overall.