How to Use Docker in Actions Runner Controller (ARC) Runners Securely

Welcome back to the second installment of our GitHub Actions Runner Controller series. In the previous post, we introduced you to the actions-runner-controller and its fundamental concepts. Today, we delve deeper into one of the most common questions DevOps/security engineers have: how to run docker securely in ARC.

ARC runs untrusted dependencies and build tools in a highly privileged environment. If your ARC cluster is not configured to run docker securely, a malicious dependency can break out the runner pod and compromise the Kubernetes host.

Did you know, your GitHub Actions environment also needs to be secured for runners to function safely? Our blog series on GitHub Actions security can help you ensure your Actions are fully secure from CI/CD threats:

‍7 GitHub Actions security best practices (with checklist)

8 GitHub Actions Secrets Management Best Practices to Follow

5 Effective Third-Party GitHub Actions Governance Best Practices

GITHUB_TOKEN: How It Works and How to Secure Automatic GitHub Action Tokens

Pinning GitHub Actions for Enhanced Security: Everything You Should Know

How does GitHub Actions use containers?

Containers are a popular way to distribute and run software as it includes everything needed to run an application. GitHub Actions uses Docker as the container platform. The GitHub hosted runner virtual machine already has docker pre-installed. There are two primary uses cases for GitHub Actions to engage with containers

Building container images

If you employ containerized workloads such as Kubernetes, you are likely building container images in GitHub Actions. There are several community-supported GitHub Actions that can be used for building container images from source code. In addition, one can also run docker commands manually to build and publish container images. We have shared a sample snippet GitHub Actions workflow to build a container image below:

jobs: 
  publish: 
    runs-on: self-hosted-arc 
    steps: 
      - name: Checkout Code 
        uses: actions/checkout@v4 
 
      - name: Build and Publish to Registry 
        uses: elgohr/Publish-Docker-Github-Action@v 
        with: 
          name: orgname/reponame/production:latest 
          username: ${{ github.actor }} 
          password: ${{ secrets.GITHUB_TOKEN }} 
          registry: ghcr.io

Running Container Image

There are three different ways for GitHub Actions to run container images. These scenarios are equivalent of GitHub Actions running “docker run” to run docker images.

GitHub Actions workflow jobs in container

By default, GitHub Actions jobs run inside a virtual machine. GitHub Actions also supports running it inside a container. This allows users to customize the runtime environment without needing to host GitHub runners. For instance, the workflow job below operates on the node container image rather than the default runner VM:

jobs: 
  container-test-job: 
    runs-on: self-hosted-arc 
    container: 
      image: node:18 
      env: 
        NODE_ENV: development 
      ports: 
        - 80 
      volumes: 
        - my_docker_volume:/volume_mount 
      options: --cpus 1 
    steps: 
      - name: Check for dockerenv file 
        run: (ls /.dockerenv && echo Found dockerenv) || (echo No dockerenv) 

Docker Container GitHub Actions

Action authors have the flexibility to create their actions using either JavaScript or Docker containers. Opting for Docker allows them to use any programming language, not just JavaScript. GitHub Actions uses Docker to run these containerized actions.

Also Read: Deploy Actions Runner Controller (ARC) using ArgoCD: A Step-by-Step Guide

Container Services

Service containers provide a streamlined way to host services essential for workflow operations or testing. For example, an integration test workflow may require access to a database or memory cache. The example below creates a service named 'redis' to run the integration-test job:

jobs: 
  integration-test: 
    runs-on: self-hosted-arc 
  ... 
    services: 
      redis: 
        image: redis 

Delving into container modes in ARC  

ARC Container modes determine how ARC runs and builds containers within the GitHub Actions workflow. To leverage containers inside a GitHub Actions workflow, we need to use one of the following container modes in ARC. These are listed in ascending order of security preference:

Least secure: Docker In Docker (dind)

This is the default container mode in ARC. Docker In Docker (dind) allows developers to run a Docker container within an already running Docker container to support CI/CD pipelines and create sandboxed container environments. ARC allows users to run dind as a docker sidecar container with the runner container or run the entire runner pod in just one container. In this case, anytime ARC runner needs to run a docker image, it runs it as a nested container inside the dind container. It can also issue `docker build` commands like a typical virtual machine host to use docker daemon to build container images.

The primary security concern with this approach is that it requires the privileged mode in Kubernetes. The Docker daemon has access to the underlying kernel of the host machine. Privileged mode allows a Kubernetes container to escape its namespace and access the entire Kubernetes host. This can compromise the entire Kubernetes cluster. There are several well-known attack vectors to abuse the privileged mode to get host access in Kubernetes. As GitHub Actions primarily run third-party code in this privileged environment, a malicious dependency or build tool can leverage the privileged mode to break out to the runner pod and persist on the Kubernetes node. This is the least secure container mode in ARC.

The following commands show that the runner pod is running in the privileged mode.

$ kubectl get runners 
NAME                        ORGANIZATION    STATUS      AGE 
akurmi-dev-org-nlqmh-sf6xr  akurmi-dev-org  Running     18s 
akurmi-dev-org-nlqmh-wmsbl  akurmi-dev-org  Running     18s 

$ kubectl get pod akurmi-dev-org-nlqmh-sf6xr -o yaml 
apiVersion: v1 
kind: Pod 
... 
    securityContext: 
      privileged: true
...

We have also prepared a container breakout PoC by a compromised GitHub Actions workflow.

Somewhat secure: Rootless Docker in Docker (dind)

When you use the default dind mode, the runner process and Docker daemon run as root inside the container. In the rootless dind mode, these processes run as a non-root user. This mode still requires the privileged Kubernetes mode, the reduced privileges reduce the risk.

As the runner process runs as a non-root user, it cannot install any build tools or dependencies on the container as part of the workflow run. Hence the organization needs to install all the required tools and dependencies in their base runner container image. This increases the operational overhead. In terms of the security, this is the second best container mode for ARC.

The following commands show that the runner pod still runs in the privileged mode.

$ kubectl get runners 
NAME                        ORGANIZATION    STATUS      AGE 
akurmi-dev-org-nlqmh-sf6xr  akurmi-dev-org  Running     18s 
akurmi-dev-org-nlqmh-wmsbl  akurmi-dev-org  Running     18s 

$ kubectl get pod akurmi-dev-org-nlqmh-sf6xr -o yaml 
apiVersion: v1 
kind: Pod 
... 
    securityContext: 
      privileged: true
...

The following screenshot shows that the Actions runner process is not able to use root in this mode.

Most secure: ARC Kubernetes Mode

In this mode, every container runs as a separate Kubernetes pod. The primary security advantage of this mode is that it does not require the privileged Kubernetes mode. In this case, the Actions runner process uses a web hook to request Kubernetes to spin up new pods for running containers inside the runner namespace as required. This is the most preferred option from a security perspective. The following command shows that the runner pod in this mode does not run in the privileged mode.

kubectl get pod arc0-lsgp4-runner-szwvj  -o yaml 
apiVersion: v1 
kind: Pod 
... 
  securityContext: {} # privileged flag is false 
... 

When a workflow requires a container to be running, ARC creates a new pod as shown below:

$ kubectl get pods  
NAME                               READY   STATUS    RESTARTS   AGE 
arc0-lsgp4-runner-szwvj            1/1     Running   0          3m47s 
arc0-lsgp4-runner-szwvj-workflow   2/2     Running   0          9s 

As the runner container does not have access to Docker daemon, it cannot build container images using docker. As an alternative, one can use alternative container image building solutions such as Kaniko and buildah to build container images. These solutions don’t need Docker daemon. For example, the following example uses buildah to build a container image.

jobs: 
  build: 
    name: Build image 
    runs-on: self-hosted-arc 
    steps: 
    - uses: actions/checkout@v3 
    - name: Buildah Action 
      uses: redhat-actions/buildah-build@v2 
      with: 
        image: my-new-image 
        tags: v1 ${{ github.sha }} 
        containerfiles: | 
          ./Containerfile 
        build-args: | 
          some_arg=some_value

Comparison

The following table summarizes the key attributes of the three container modes we discussed above.

Conclusion

The container mode you select in ARC can have a profound effect on the security of your ARC cluster. In this blog post, we explore three container modes supported by ARC.

If you would like us to cover a specific ARC topic, please reach out to us.

Breaking out of the pod container is not the only security risk with ARC. A compromised dependency or build tool can exfiltrate source code and CI/CD secrets from any GitHub Actions runners. We recommend implementing runtime CI/CD security to prevent such threats. If you're looking to implement runtime security for your CI/CD pipelines and enhance your security, StepSecurity can help you out. Get in touch with us to explore how we can enhance your CI/CD security.