Introduction
Managing Actions Runner Controller (ARC) deployments can be a tedious task, especially when you're dealing with upgrades and maintenance tasks. In this blog post, we'll cover how to deploy Actions Runner Controller using ArgoCD—a popular Kubernetes GitOps tool.
What is Actions Runner Controller (ARC)?
ARC is a Kubernetes operator designed to manage and scale self-hosted GitHub Action runner pods. It is one of the go-to options for running GitHub Actions workflows in a self-hosted environment. Please refer to our previous blog posts in the series to learn more about Actions Runner Controller:
Introduction to GitHub Actions Runner Controller: A Blog Series
How to Use Docker in Actions Runner Controller (ARC) Runners Securely
Looking to secure your ARC environments? Our eBPF-powered Kubernetes native solution on AWS EKS, GCP GKE, and Azure AKS is made just for that! Explore how you can fortify your ARC cluster with us.
Why Use ArgoCD?
ArgoCD is a declarative, GitOps Continuous Delivery (CD) tool for Kubernetes. It is an open-source project typically used in conjunction with a Continuous Integration (CI) platform such as GitHub Actions to manage Kubernetes deployments. With ArgoCD, your Kubernetes resources are version-controlled and can be automatically updated to match the state in your Git repository. This makes it an ideal choice for deploying and managing ARC.
Prerequisites
Before you proceed, ensure you have the following setup:
- A Kubernetes cluster for hosting ARC and ArgoCD
- kubectl configured to manage the above cluster
- Helm CLI
We assume that you will be deploying ArgoCD first in the Kubernetes cluster and then use it to manage ARC resources.
Setting up ArgoCD and sealed-secrets
Deploy ArgoCD
To deploy ArgoCD, run the following command as per the official documentation
We will be using the ArgoCD CLI which you can install on your machine by following these steps.
Deploy sealed-secrets
For managing ARC’s GitHub Personal Access Token (PAT) (more on this in the next section) and other secrets, we will use sealed-secrets.
For production, generate a key pair, store it securely offline, and configure sealed-secrets to use this key. This way, you won’t have to change your encrypted secrets in your Git repository every time you rebuild your Kubernetes cluster.
Generate a key pair
You should perform this step only once and store private.key and certificate.crt securely. For subsequent installations, skip this step and pass private.key and certificate.crt that you generated previously to the next step.
Install sealed-secret
Run the following Helm command to install sealed-secrets.
ArgoCD Configuration
CLI
To access the ArgoCD API server, run the following kubectl port forwarding command in a terminal.
Run the following commands to configure argocd CLI to use the ArgoCD instance:
Web UI
For this installation guide, you don’t need to use the web UI. Access the ArgoCD web interface at localhost:8080 to track progress or troubleshoot issues.
GitHub Repository for ArgoCD applications
If you don’t already have a GitHub repository for hosting ArgoCD applications, create one in your GitHub organization. For better security, this must be a private repository. We have shared the ArgoCD apps that will be uses for the blog post at https://github.com/step-security/code-samples/tree/main/deploy-arc-using-argocd
Personal Access Token (PAT) to authorize ArgoCD
As the ArgoCD repository that you created above is private, ArgoCD would require a GitHub PAT to access the repository. Follow these steps:
- Create a new GitHub bot user named argocd
- Give the bot user read access to the ArgoCD repository
- Create a GitHub PAT for the bot user with repo permissions as shown in the image below:
You can also create an equivalent fine-grained PAT with your personal account.
Onboard the GitHub repository onto ArgoCD
Deploy Actions Runner Controller using ArgoCD
Create a GitHub Personal Access Token for ARC
In order to authorize ARC to access relevant GitHub resources, you will need a GitHub Personal Access Token with appropriate permissions. Ideally, create a new GitHub bot account for ARC, authorize it to access the GitHub repositories/organizations/enterprises you need, and store this safely as you'll need it later.
Another way to authorize ARC is to create a GitHub App and provide the private key to ARC. One advantage of using a GitHub App over a PAT is that the App has higher API throttling limits.
You can read more about ARC-supported GitHub authentication models and required PAT/GitHub App permissions here.
Setting Up the ARC ArgoCD App of Apps
ArgoCD App of Apps consists solely of other apps within ArgoCD. To simplify ARC installation, we have authored the following ARC App of Apps that installs all the required components to build a functional ARC cluster. Please clone the repository into your ArgoCD repository before proceeding forward.
There are two flavors of ARC: community-supported ARC and GitHub-supported ARC. You can learn about how these flavors differ from each other in our ARC series introductory blog post. Please follow one of the following sections depending on which flavor you are using.
In both models, we create two ArgoCD apps, one for the controller and one for hosting runner pods. The advantage of having a separate ArgoCD app for runners is that one can create multiple runners using the same ArgoCD app definition in GitHub by passing different parameters.
Community-Supported ARC
Sealed secret
1. Create github_token_secret.yml with the GitHub PAT in the following format
2. Run the following command to create a new file named github_token_sealed_secret.yml with the sealed secret.
3. Replace spec.encryptedData.github_token in community-supported/controller/templates/pat-secret.yaml with the content of spec.encryptedData. github_token from github_token_sealed_secret.yml.
ARC Controller
1. Copy the app into the ArgoCD repository you created above and run the following argocd CLI command
ARC Runners
As ARC runners depend on the ARC controller, it cannot be created as part of the ARC App of Apps. Furthermore, having a separate ArgoCD application for ARC Runners gives us the ability to deploy multiple runners if required.
In the GitHub runner ArgoCD app, we are not using any runner labels. The runner is also not part of a specific runner group. You can update the ArgoCD app with these attributes if required. Define runner configurations inside the values directory. For example, we have created step_security.yaml to create a runner for the step-security organization.
Check pods status
At this point, if you list all running pods, you will see the ARC controller and runner pods in running state.
GitHub-supported ARC
ARC Controller
Deploy Actions Runner Controller by running the following command
sealed-secret
With the community-supported model, the ARC controller had access to the GitHub PAT. Whereas in this model, you need to grant runners access to a GitHub PAT instead.
1. Create github_token_secret.yml with the GitHub PAT in the following format.
2. Run the following command to create a new file named github_token_sealed_secret.yml with the sealed secret.
3. Update deploy-arc-using-argocd/github-supported/runners/templates/pat-secret.yaml with the value of sealed secret in github_token_secret_sealed.yml.
ARC Runner
Just like the community-managed ARC flavor, the ArgoCD app for runner pods does not use any runner labels. The runner is also not part of a specific runner group. You can update the ArgoCD app with these attributes if required. Deploy ARC runner by running the following command
Check Pod Status
Run the following command to confirm that all Kubernetes pods are in running state.
Troubleshooting
Running argocd CLI commands on argocd server
We have frequently seen Kubernetes port forwarding dropping packets with the following error:
If you consistently see these errors, another way to run ArgoCD CLI commands is to use the ArgoCD Kubernetes pod itself. To use this method, you would need to create a new ArgoCD user, generate a token, and use that token for running ArgoCD commands. We have shown the steps to run ArgoCD CLI commands on the ArgoCD Kubernetes pod below.
Also Read: Secure your Actions Runner Controller (ARC) Environment using StepSecurity
Happy Deploying!
A compromised dependency or build tool can exfiltrate source code and CI/CD secrets from any GitHub Actions runners. We recommend implementing runtime CI/CD security to prevent such threats. To know more about how you can do that, get in touch with our team.
