CISA Monitors Network Egress Traffic and Hardens their GitHub-hosted Runners

Introduction

CISA (Cybersecurity and Infrastructure Security Agency) provides regional cyber and physical services to support security and resilience across the United States. They have over 400 repositories on GitHub ranging from monitoring solutions, domain-name registrars, assessment automation, etc. CISA has contributed greatly to the industry and has enabled developers, security teams, and DevOps teams to secure their workflows from evolving threats.

The Challenge

CISA’s open-source projects use GitHub Actions CI/CD pipelines. CI/CD pipelines are attractive targets for malicious cyber actors, as evidenced by the increasing compromises over time, such as the SolarWinds and Codecov breaches.

NSA and CISA have authored guidance titled "Defending Continuous Integration/Continuous Delivery (CI/CD) Environments" to provide recommendations and best practices for hardening CI/CD pipelines against attackers to secure DevSecOps CI/CD environments. This guidance recommends implementing network segmentation and traffic filtering and endpoint detection and response (EDR) tools in the CI/CD environment. You can read more about the guidance below.

https://www.stepsecurity.io/blog/stepsecuritys-alignment-with-cisas-ci-cd-security-guidance

Traditional network security solutions and EDR tools don’t work well in CI/CD environments due to the ephemeral nature of CI/CD runners (build servers). Moreover, the existing tools lack context about CI/CD pipelines and are not built to detect CI/CD specific attack patterns.

The Solution: StepSecurity Harden-Runner

StepSecurity Harden-Runner is a network and runtime security solution for GitHub-hosted and self-hosted runners. The solution has been designed to prevent attacks on the CI/CD pipelines by monitoring files, processes, and network activity on the runners (build servers). This enables it to detect compromised dependencies and poised workflows that aim to exfiltrate CI/CD credentials or tamper with source code or artifacts during the build process.

CISA leverages Harden-Runner to:
1. Detect anomalous egress traffic at the DNS (Layer 7) and network layers (Layers 3 and 4) to prevent exfiltration of CI/CD credentials.
2. Gain instant contextualized insights into network and file events of each step of the workflow.
3. Detect if source code is being tampered during the build process to inject a backdoor

Here’s what one of the developers from CISA using Harden-Runner tosecure their workflows had to say about it:

“Harden-Runner GitHub Action is being configured to run in audit mode. Itshould warn us if an Action is reaching out to an unexpected web address,overwriting source code, etc”

Software Developer
Cybersecurity and Infrastructure Security Agency (CISA)

Easy Onboarding

For GitHub-hosted runners, onboarding is as easy as adding the Harden-Runner GitHub Action as the first step in each workflow. Once that is done, each workflow run is monitored by Harden-Runner and the insights are shown on the StepSecurity dashboard. Due to the ease of onboarding, workflows across 175 public repositories in the CISA GitHub organization use Harden-Runner.

Here is an example of a workflow where Harden-Runner is being used.

https://github.com/cisagov/teamserver-packer/actions/runs/7390517306/workflow#L41-L44

Granular Insights and Detections

The insights for the workflow runs are shown on the StepSecurity dashboard. These insights are public for public repositories. For private repositories, the insights URL is not public, and needs authorization to access.

Here is an example of the insights for a workflow in one of CISA’s public repositories.

https://app.stepsecurity.io/github/cisagov/teamserver-packer/actions/runs/7390517306?nav=1

A baseline is created for the outbound traffic for each job, and if a process in a future run reaches out to a destination not in the baseline, an anomalous network call detection is created. Here is an example, where the job reached out to a new destination (shown in orange).

Similarly, a baseline is created for file overwrites during the build process, and if a new file is overwritten after the baseline is created, a file overwrite detection is triggered. In this case, the file XYZ is being overwritten as part of the build process.

These insights and detections are critical to detect potentially compromised dependencies and workflows and for forensic purposes.

Report of All outbound destinations across the Organization

In addition to a workflow level view, a report is generated with a list of all network destinations at the GitHub Organization level. This enables developers or security teams to review the list periodically and if a destination is suspicious, they can view the workflow runs that made the outbound call to investigate further. The report is public for public repositories and for CISA’s public repositories, it can be found here:

https://app.stepsecurity.io/github/cisagov/actions/all-endpoints

Impact

StepSecurity Harden-Runner enables CISA to monitor network egress traffic with granular job-level baselines and hardens their GitHub Actions workflows to protect against CI/CD threats. Its ease of use has enabled StepSecurity Harden-Runner to be used across hundreds of CISA’s public repositories. Harden-runner solves the need for CISA’s public repositories for network traffic monitoring and endpoint detection and response (EDR) in the CI/CD environment.