Microsoft Leverages StepSecurity to Secure CI/CD for their Open-Source Projects

Introduction

With about 6,000 repositories and more than 5,000 people involved, Microsoft has made huge contributions to the open-source community on GitHub. Their popular projects include Visual Studio Code, TypeScript, PowerToys, and Windows Terminal, with which they have enabled thousands of developers to maximize their productivity.

The Challenge

Preventing Supply Chain Attacks by Detecting Compromised Workflows in CI/CD Pipelines

In January 2024, security researchers successfully carried out a supply chain attack on PyTorch and many other organizations, including GitHub itself, by exploiting CI/CD vulnerabilities in their repositories.

Another similar incident took place in December 2020 when a security researcher broke into Microsoft’s Visual Studio Code GitHub repository. The attack was due to a vulnerability in the CI script, and the researcher was able to get write access to the repository.

Incidents like these have highlighted how important it is to secure GitHub Actions.

Implementing Recommended GitHub Actions Security Best Practices

With thousands of repositories and developers worldwide using their projects and tools on GitHub, Microsoft needs to ensure that their workflows are secure and have GitHub Actions security best practices well-integrated. Now, manually implementing various security best practices in so many projects is quite a laborious task for Microsoft developers and demands a lot of their time which could be otherwise spent in innovative and productive tasks.

Orchestrating Standardized Workflows Across Multiple Repositories

To keep their workflows in check, scan vulnerable dependencies and maintain the code quality, Microsoft needs to add OpenSSF Scorecard, Static Application Security Testing (SAST) and Dependency review (SCA) workflows for their projects.

Since manual integration of each one of these in their projects is a laborious task, they need to fast track it and automate this task for their developers.

The Solution: StepSecurity GitHub Actions Security Platform

Hardening of GitHub-hosted Runners

To detect compromised workflows and dependencies and harden their runners, Microsoft leverages StepSecurity Harden-Runner. Harden-runner hardens GitHub-hosted and self-hosted runners by providing outbound network controls and runtime security.

The following is an example of a workflow from Microsoft using Harden-Runner in audit mode. In audit mode, Harden-Runner audits outbound calls at the DNS, HTTPS, and network layers.

https://github.com/microsoft/msquic/blob/main/.github/workflows/docker-publish.yml#L28-L31

The build log contains a link to the StepSecurity dashboard with network events for that workflow run. The dashboard is public for open-source projects and private for private repositories.

https://app.stepsecurity.io/github/microsoft/msquic/actions/runs/7808049560

The outbound calls for each job are aggregated, and based on past runs,a block policy is recommended. Developers can then update their workflow to use Harden-Runner in block mode, which blocks any outbound calls not in the allowed list in real-time.

Here is an example of a workflow in a Microsoft project that is configured to use the block mode.

https://github.com/microsoft/ebpf-for-windows/blob/main/.github/workflows/update-docs.yml#L32-L38

The StepSecurity dashboard also includes a report of all the destinations to which outbound calls have been made across the workflows in a GitHub organization. This makes it easy to review the outbound calls periodically and find workflows that made a particular call. Here’s what the reports look like for the Microsoft and the Azure GitHub Organizations.

All outbound endpoints for the Microsoft GitHub Organization:
https://app.stepsecurity.io/github/microsoft/actions/all-endpoints

All outbound endpoints for the Azure GitHub Organization:
https://app.stepsecurity.io/github/Azure/actions/all-endpoints

Implementing GitHub Actions Security Best Practices Automatically with StepSecurity Orchestration solution

Microsoft resolves the challenge of implementing GitHub Actions security best practices in various projects with StepSecurity’s Orchestration solution using https://app.stepsecurity.io/securerepo. The orchestration solution not only recommends security fixes but also creates automatic pull requests to implement them in their projects.

Some GitHub Actions best practices implemented by Microsoft with StepSecurity’s help include pinning actions to full-length commit SHA, setting minimum token permissions, integrating Dependabot, and integrating pre-commit hooks.

Here are some pull requests automatically generated by StepSecurity for Microsoft’s projects:

Orchestrating Standardized Workflows Across Repositories

To ensure their projects have standardized workflows, Microsoft leverages StepSecurity to add missing workflows to their projects using automatic pull requests.

Here are some automated pull requests generated by StepSecurity to standardize Microsoft’s workflows to add the OpenSSF Scorecard, Static Application Security Testing (SAST) and Dependency Review (SCA) workflows:

Community Feedback

StepSecurity is widely used in Microsoft repositories and is recommended by developers from Microsoft to secure GitHub Actions. Here is a recommendation by a Senior Software Engineer at Microsoft for StepSecurity:

The Impact

With StepSecurity’s platform, Microsoft secures its GitHub Actions workflows at scale from CI/CD threats and supply chain attacks. Further, StepSecurity enables Microsoft developers to not only implement CI/CD security, but also save precious hours in doing monotonous tasks by automating implementation of GitHub Actions security best practices.

StepSecurity has saved approximately 120 developer hours for Microsoft. Microsoft developers can now attain standardized workflows, maintain code quality, and ensure secure workflows with StepSecurity’s orchestration platform with minimal effort. Finally, due to the self-serve nature of the platform, Microsoft developers are able to use the platform independently with no prior training.