GitHub Actions is a CI/CD Platform that allows you to automate your build, test, and deployment pipeline. GitHub Actions jobs are run in the cloud by default, but you may want to run your jobs in your environment. Self-hosted runner can be used for such use cases but requires the provisioning and configuration of a virtual machine instance. Instead, if you already have a Kubernetes cluster, it makes more sense to run the self-hosted runner on top of it.
Actions Runner Controller (ARC) makes that possible. ARC is a Kubernetes operator that orchestrates and scales self-hosted runners for GitHub Actions.
While ARC makes it easy to set up self-hosted runners for GitHub Actions, it is the customer’s responsibility to secure the cluster and the ARC CI/CD Environment. GitHub Actions executes third party code, which may not be trustworthy, in a privileged environment. As a result, a compromised or poisoned workflow can exfiltrate CI/CD credentials or tamper with release builds.
If you are looking to secure your Actions Runner Controller (ARC) environment, get in touch with us.
As an example, in the Codecov incident, CI/CD secrets were exfiltrated from thousands of build servers for over two months without being detected. And in the SolarWinds incident, a source code file was tampered in the CI/CD pipeline to inject a backdoor.
This article shows how you can defend against such CI/CD attacks in your ARC environment using StepSecurity’s GitHub Actions Security Platform.
Runtime Security for Actions Runner Controller (ARC) Environment
StepSecurity is a Kubernetes-aware security observability and runtime enforcement platform purpose built for ARC. It uses eBPF, allowing for reduced observation overhead and real-time enforcement of policies to prevent CI/CD attacks.
Security Observability
Once you deploy StepSecurity on your ARC Kubernetes cluster, you get security visibility across all of your GitHub Actions workflow runs. It does not require any code or repository changes. You get insights into the network and file events associated with each step of the GitHub Actions workflow runs.
As an example, here is a screenshot that shows outbound network calls made by each step of a workflow run. You can click on the step in the dashboard to go to the build log for that step.
Using the StepSecurity dashboard, you can click on any workflow run, and view the security insights for that run.
Prevent Exfiltration of Code and CI/CD Credentials
If a GitHub Actions workflow is poisoned, either due to compromise of a developer credential, or due to a compromised dependency or a build tool, it can be used to exfiltrate code or CI/CD credentials. The CI/CD credentials are typically high privileged and may give an attacker access to the cloud environment.
In the insights page for each workflow run, you will notice a recommended policy to block outbound traffic to allowed destinations. This policy is recommended based on the outbound calls previously made for a given workflow. Since a workflow typically needs to download code, dependencies, and publish artifacts to the cloud, the outbound traffic from each workflow is predictable and does not change often.
You can set the recommended policy in your workflow by adding the StepSecurity Harden-Runner GitHub Action. Once you do this, any outbound calls that are not in the allowed list will be blocked! Not just that, you will also receive a notification for such block events. The policy is set at the DNS (Layer 7) and Network (Layers 3 and 4) layers. So, you just see a list of allowed endpoints in terms of the domain names and ports to which the traffic should be allowed.
Here is an example where the Codecov breach is being simulated in a workflow that has a block policy.
As you can see from the workflow run, and the insights page, the outbound call was blocked. This is an important feature to prevent CI/CD attacks in your ARC environment.
You can see a real example of this in the StepSecurity GitHub Actions Goat project, which is a deliberately vulnerable GitHub Actions Environment to learn about GitHub Actions Security. Here is a workflow run on a self-hosted ARC environment with a block policy. Here you can view the security insights for the run. As you can see the outbound call that was not in the allowed list has been blocked.
Detect Tampering of Release Builds
If in any of the jobs, source code files or artifacts are tampered during the GitHub Actions workflow, you will see an event for it in the insights view. Such overwrite of files are not expected in a typical release build. You can choose to be notified of these detections via an email or Slack. You can then investigate if this overwrite is a false positive or if it needs more investigation.
Here is a simulation of the SolarWinds incident. A source code file is being tampered during the build. You can see from the screenshot below how this is detected.
You can see a real example of this in the StepSecurity GitHub Actions Goat project. Here is a workflow run on a self-hosted ARC environment being monitored by StepSecurity, and here you can view the security insights for the run. As you can see the suspicious file event has been detected without making any changes to the workflow file.
Runtime Security for GitHub-Hosted Runner
While this article focusses on runtime security for ARC Environment, the Harden-Runner GitHub Action provides the same set of features for GitHub-hosted runners as well. This means that you can easily change from GitHub-hosted to ARC runners, and the runtime security will work seamlessly across the two environments.
Harden-Runner GitHub Action is being used by over 1,200 open-source projects, including projects from industry giants like Microsoft, Google, and DataDog. You can explore how these projects are using the Action in the Harden Runner GitHub Action repository.
Conclusion
In this article, we looked at the key threats in an Actions Runner Controller (ARC) Environment – of tampering of release builds and exfiltration of code or CI/CD credentials. We then looked at how Runtime Security can be implemented for the ARC cluster using StepSecurity. We looked at Security Observability, setting Runtime policies, and Runtime enforcement of those policies to prevent common CI/CD attacks. We then saw that the same solution can also be used for GitHub-hosted runners. If you are interested in this solution to secure your ARC environment or to use it for your GitHub-hosted runners, checkout the StepSecurity GitHub Actions Platform at https://stepsecurity.io or click on the button below.
Introduction
GitHub Actions is a CI/CD Platform that allows you to automate your build, test, and deployment pipeline. GitHub Actions jobs are run in the cloud by default, but you may want to run your jobs in your environment. Self-hosted runner can be used for such use cases but requires the provisioning and configuration of a virtual machine instance. Instead, if you already have a Kubernetes cluster, it makes more sense to run the self-hosted runner on top of it.
Actions Runner Controller (ARC) makes that possible. ARC is a Kubernetes operator that orchestrates and scales self-hosted runners for GitHub Actions.
While ARC makes it easy to set up self-hosted runners for GitHub Actions, it is the customer’s responsibility to secure the cluster and the ARC CI/CD Environment. GitHub Actions executes third party code, which may not be trustworthy, in a privileged environment. As a result, a compromised or poisoned workflow can exfiltrate CI/CD credentials or tamper with release builds.
If you are looking to secure your Actions Runner Controller (ARC) environment, get in touch with us.
As an example, in the Codecov incident, CI/CD secrets were exfiltrated from thousands of build servers for over two months without being detected. And in the SolarWinds incident, a source code file was tampered in the CI/CD pipeline to inject a backdoor.
This article shows how you can defend against such CI/CD attacks in your ARC environment using StepSecurity’s GitHub Actions Security Platform.
Runtime Security for Actions Runner Controller (ARC) Environment
StepSecurity is a Kubernetes-aware security observability and runtime enforcement platform purpose built for ARC. It uses eBPF, allowing for reduced observation overhead and real-time enforcement of policies to prevent CI/CD attacks.
Security Observability
Once you deploy StepSecurity on your ARC Kubernetes cluster, you get security visibility across all of your GitHub Actions workflow runs. It does not require any code or repository changes. You get insights into the network and file events associated with each step of the GitHub Actions workflow runs.
As an example, here is a screenshot that shows outbound network calls made by each step of a workflow run. You can click on the step in the dashboard to go to the build log for that step.
Using the StepSecurity dashboard, you can click on any workflow run, and view the security insights for that run.
Prevent Exfiltration of Code and CI/CD Credentials
If a GitHub Actions workflow is poisoned, either due to compromise of a developer credential, or due to a compromised dependency or a build tool, it can be used to exfiltrate code or CI/CD credentials. The CI/CD credentials are typically high privileged and may give an attacker access to the cloud environment.
In the insights page for each workflow run, you will notice a recommended policy to block outbound traffic to allowed destinations. This policy is recommended based on the outbound calls previously made for a given workflow. Since a workflow typically needs to download code, dependencies, and publish artifacts to the cloud, the outbound traffic from each workflow is predictable and does not change often.
You can set the recommended policy in your workflow by adding the StepSecurity Harden-Runner GitHub Action. Once you do this, any outbound calls that are not in the allowed list will be blocked! Not just that, you will also receive a notification for such block events. The policy is set at the DNS (Layer 7) and Network (Layers 3 and 4) layers. So, you just see a list of allowed endpoints in terms of the domain names and ports to which the traffic should be allowed.
Here is an example where the Codecov breach is being simulated in a workflow that has a block policy.
As you can see from the workflow run, and the insights page, the outbound call was blocked. This is an important feature to prevent CI/CD attacks in your ARC environment.
You can see a real example of this in the StepSecurity GitHub Actions Goat project, which is a deliberately vulnerable GitHub Actions Environment to learn about GitHub Actions Security. Here is a workflow run on a self-hosted ARC environment with a block policy. Here you can view the security insights for the run. As you can see the outbound call that was not in the allowed list has been blocked.
Detect Tampering of Release Builds
If in any of the jobs, source code files or artifacts are tampered during the GitHub Actions workflow, you will see an event for it in the insights view. Such overwrite of files are not expected in a typical release build. You can choose to be notified of these detections via an email or Slack. You can then investigate if this overwrite is a false positive or if it needs more investigation.
Here is a simulation of the SolarWinds incident. A source code file is being tampered during the build. You can see from the screenshot below how this is detected.
You can see a real example of this in the StepSecurity GitHub Actions Goat project. Here is a workflow run on a self-hosted ARC environment being monitored by StepSecurity, and here you can view the security insights for the run. As you can see the suspicious file event has been detected without making any changes to the workflow file.
Runtime Security for GitHub-Hosted Runner
While this article focusses on runtime security for ARC Environment, the Harden-Runner GitHub Action provides the same set of features for GitHub-hosted runners as well. This means that you can easily change from GitHub-hosted to ARC runners, and the runtime security will work seamlessly across the two environments.
Harden-Runner GitHub Action is being used by over 1,200 open-source projects, including projects from industry giants like Microsoft, Google, and DataDog. You can explore how these projects are using the Action in the Harden Runner GitHub Action repository.
Conclusion
In this article, we looked at the key threats in an Actions Runner Controller (ARC) Environment – of tampering of release builds and exfiltration of code or CI/CD credentials. We then looked at how Runtime Security can be implemented for the ARC cluster using StepSecurity. We looked at Security Observability, setting Runtime policies, and Runtime enforcement of those policies to prevent common CI/CD attacks. We then saw that the same solution can also be used for GitHub-hosted runners. If you are interested in this solution to secure your ARC environment or to use it for your GitHub-hosted runners, checkout the StepSecurity GitHub Actions Platform at https://stepsecurity.io or click on the button below.
.png)
