Case Study

Google Automates GitHub Actions Security for their Open-Source Projects with StepSecurity

Google Automates GitHub Actions Security for their Open-Source Projects with StepSecurity

This case study talks about how Google leverages StepSecurity’s GitHub Actions security platform to harden their GitHub-hosted runners and automate various GitHub Actions security best practices in several of their open-source projects.

Runners:

Github-Hosted

Introduction

Google has greatly contributed to the developer community over the years with its open-source projects on GitHub. These projects include machine learning frameworks, cloud computing tools, and more. With its projects, Google has not only empowered thousands of developers worldwide but has also proved its dedication to knowledge-sharing with the community.

The Challenge

Detecting Compromised Workflows to Secure CI/CD Pipeline from Supply Chain Attacks

CI/CD runs untrusted third-party software in a highly privileged environment that has access to source code, software builds, and elevated credentials. This can lead to supply chain attacks causing massive losses to enterprises.

This happens when compromised workflows, dependencies, and build tools exfiltrate CI/CD credentials, or tamper source code, dependencies, or artifacts during the build. Google needed to harden the GitHub-hosted runners on which their pipelines run to detect compromised dependencies and poisoned workflows.

Implementing Recommended GitHub Actions Security Best Practices

Another major challenge faced by Google was compliance with GitHub Actions’ security best practices. There are various security misconfigurations in GitHub Actions and addressing each one of them for so many projects was turning out to be quite a laborious task for Google's developers. Google needed a solution that would ease this process and enable their developers to save time and be more productive.

Orchestrating Standardized Workflows Across Multiple Repositories

Google needed to add OpenSSF Scorecard, Static application security testing (SAST) and Dependency review (SCA) workflows for their projects. With so many projects, it was a challenge for them to consistently add the same manually in each.

The Solution: StepSecurity GitHub Actions Security Platform

Network and Runtime Security with StepSecurity Harden-Runner

Google leverages StepSecurity Harden-Runner to harden the GitHub-hosted runners on which their workflows run. Harden-Runner provides network and runtime security for GitHub-hosted and self-hosted runners. This purpose-built platform is based on the learnings from past CI/CD security attacks (e.g., the SolarWinds and Codecov incidents).

Harden-Runner provides contextualized runtime security insights for Google’s projects’ workflows, prevents exfiltration of code and CI/CD credentials by blocking egress traffic, detects if there is source code tampering, and detects deviations from the usual runtime behavior in workflows.

https://github.com/GoogleCloudPlatform/functions-framework-ruby/blob/main/.github/workflows/lint.yml#L18

sample google github actions workflow using stepsecurity harden-runner

The insights from Harden-Runner are available on the StepSecurity dashboard, which is public for open-source projects and private for private repositories.

https://app.stepsecurity.io/github/GoogleCloudPlatform/functions-framework-ruby/actions/runs/7450585331

Harden-Runner insights for a workflow run

The insights are aggregated at the GitHub organization level making it easy to spot suspicious outbound calls across workflows that are using Harden-Runner. The “All Observed Endpoints” report shows all the unique destinations to which outbound network calls were made. Each destination has links to some of the workflows that made that call. This enables the security team to review the list periodically and if a destination is suspicious, they can view the workflow runs that made the outbound call to investigate further.

https://app.stepsecurity.io/github/GoogleCloudPlatform/actions/all-endpoints

aggregated endpoints for the google organization

Implementing GitHub Actions Security Best Practices Automatically with StepSecurity Orchestration solution

Google implements GitHub Actions best practices with the help of StepSecurity’s orchestration solution. The solution not only recommends security fixes but also creates pull requests automatically with the recommended fixes.

Some GitHub Actions security best practices adhered to by Google withthe help of StepSecurity were limiting GitHub token permissions, hardeningrunners (with StepSecurity Harden-Runner), pinning actions to full-lengthcommit SHA, and configuring Dependabot version updates.

StepSecurity is also recommended by Google’s security team to other developers as seen in this issue:

Orchestrating Standardized Workflows Across Repositories

With the orchestration solution, Google could adopt standardized workflows across various repositories, and this further helped them to save hundreds of developer hours and enabled increased productivity.

This included using security tools like OpenSSF Scorecard, CodeQL (for SAST), and Dependency Review (SCA) in their workflows across different repositories to streamline their processes efficiently and uniformly release their software.

Here are some examples of Google’s standardized workflows enabled by StepSecurity:

Community Feedback

StepSecurity’s solutions have not only been leveraged but have also been recommended by various developers at Google. Here's one of these developers recommending the use of StepSecurity for hash pinning in an issue:

https://github.com/google/double-conversion/issues/197#issuecomment-1557835283

google engineer recommending stepsecurity

The Impact

Google's use of Harden-Runner and the StepSecurity orchestration solution enabled them to secure their workflows from possible CI/CD supply chain attacks and helped them save hundreds of developer hours in implementing GitHub Actions security best practices. This empowered developers to enhance the security of their workflows with network monitoring, detecting tampering of source code, adopting standardized workflows, and efficiently utilizing their time on innovative tasks rather than spending hours on the laborious task of abiding by security best practices. Moreover, StepSecurity Harden-Runner and orchestration solution enabled several Google developers working on different projects resolve their security problems seamlessly without any training.

‍