GitHub Actions is the Alfred to your Batman. When you don’t feel like doing something or simply don’t have the capacity to handle various tasks, you can rely on GitHub Actions to automate workflows. You can take GitHub Actions to the next level by self-hosting runners, though. But should you? Let’s find out!
We’ve been managing dyrector.io’s code on GitHub for more than a year now. One thing we’ve always struggled with was slow GitHub Actions workflows. Here’s why we’ve been contemplating switching to self-hosting our GitHub Runners.
The default GitHub runner takes longer to execute as it initializes an ephemeral runner for each job in a workflow from scratch. This method, chosen by GitHub for its simplicity and security, has its merits. Compared to this, self-hosted runners remain active, bypassing the initialization phase for every job, thus providing quicker execution. This continuous availability demands proper management to ensure subsequent runs are not interfered with by remnants from previous executions.
GitHub-hosted runners run on Ubuntu with a 2-core CPU, limiting parallel job executions to four. In a self-hosted scenario, we have the liberty to choose other OSes. We opted for Rocky Linux over Ubuntu for its open-source, enterprise-grade, and 100% Red Hat compatibility. This choice also allowed us to define the VM’s hardware parameters like CPU, memory and disk type/size. However, this freedom comes at the cost of increased maintenance overhead.
Debugging is more challenging on GitHub-hosted runners as only error messages and logs are retained. In the meantime, self-hosted runners keep everything in the “_work” and “_diag” directories, allowing real-time monitoring to understand precisely what is happening and the resources being consumed, as the running VM is under our control.
As we look into the future and explore opportunities for further improvement:
The default GitHub hosted runner functionality is free and comes with autoscaling if we look at the submitted parallel pull requests, so you don’t have to do anything for them, they are simply there and doing their job. We obviously lose this default behavior if we go on the self-hosted route.
The initial setup requires a learning curve, and maintaining the runners can demand a fair share of time. It is not so much the setting of the runners themselves, but rather the maintenance, updating, securing the VM(s) and the correct initial setting of the workflow to manage the clean up and teardown side steps for every job and job step, if necessary.
Self-hosted runners may expose your environment to potential security risks if not configured and managed properly. Something even GitHub recommends in its official docs is to use self-hosted runners with private repositories. Here’s a more detailed description about security measures for GitHub runners.
Ensure your system meets GitHub’s minimum requirements, which include:
We used a larger machine with the following specifications:
The upgrade was necessary due to the combined temporary space needs of our code, node.js, and about 10 docker containers, including a playwright container for testing. Our runners resided on an additional data disk, leaving about 8 GB free on the system disk.
Instead of using multiple small VMs with one runner each, we chose to use one large VM hosting several parallel runners. This approach minimizes VM maintenance overhead and is designed to efficiently handle multiple parallel GitHub pull requests.
Future scaling is straightforward as setting up additional runners and/or VMs is not complicated; runners distribute workflow jobs based on common labels regardless of their VM location.
We set up our self-hosted runners with these steps, here we will show actions-runner-001, but it was done in a similar way for our runners 002, 003 and so on.
At your GitHub repository’s Settings, in the left sidebar click Actions, then click Runners and finally click New self-hosted runner. Select the OS image and architecture of your self-hosted runner machine. In our case it is Linux and x64.
# Create a folder
$ mkdir actions-runner-001 && cd actions-runner-001
# Download the latest runner package
$ curl -o actions-runner-linux-x64-2.311.0.tar.gz -L https://github.com/actions/runner/releases/download/v2.311.0/actions-runner-linux-x64-2.311.0.tar.gz
# Optional: Validate the hash
# On Rocky Linux you may need to install shasum once for this validation
$ sudo dnf update
$ sudo dnf install -y perl-Digest-SHA
$ echo "29fc8cf2dab4c195bb147384e7e2c94cfd4d4022c793b346a6175435265aa278 actions-runner-linux-x64-2.311.0.tar.gz" | shasum -a 256 –c
# Extract the installer
$ tar xzf ./actions-runner-linux-x64-2.311.0.tar.gzWe only need to do this step once per VM, not per runner. You can skip this step if your OS already contains these dependencies, but for Rocky Linux 9.2 it was necessary.
# Install dependencies (on Rocky Linux dotnet core 6 was missing by default)
$ sudo ./bin/installdependencies.shWe also installed node.js, go and docker on our VM for our workflow, but these are not runner dependencies, so we will not go into detail about that here.
# Create the runner and configure it
$ ./config.sh --url https://github.com/dyrector-io/dyrectorio --token <RUNNER_TOKEN>During the configuration process, you can keep most settings at their default values, but we chose to make our runners easily identifiable by giving them unique names and adding extra labels. Initially, the configuration script provides a common name, but our objective was to test multiple runners on a single VM.
By default, a runner is tagged with three labels for Linux x64: self-hosted, Linux, and X64. However, you have the flexibility to specify additional labels during the initial configuration or later on the GitHub repository website. Unlike the default labels, you can add or remove these custom labels at any time. These labels come in handy for targeting specific groups of runners or individual runners within your workflow.
Pre-job script is not mandatory if you do not want to use it, but we need it. In the runner directory just create a .env file with this content:
ACTIONS_RUNNER_HOOK_JOB_STARTED=pre-job-script.sh
And in the pre-job bash script file you can use your additional VM specific logic which will run before every job. Important to write “exit 0” at the end of the script file, because this means the script run without errors – otherwise or if you return any other value the runner will skip this job. You can also use this to your advantage for pre checks.
You can start the runner with its run script ($ ./run.sh), but we want to run it as a service so first need to install the service and on Rocky Linux we also need to set the SELinux security context for the runsvc.sh file to ensure it operates correctly within the SELinux security policy (otherwise it will be blocked). We only need to set SELinux context and service install once.
# Set SELinux context for the runsvc script to s0 (standard security level)
$ sudo chcon system_u:object_r:usr_t:s0 runsvc.sh
# Install the runner as a service
$ sudo ./svc.sh installNow we can use the service with its start, stop, status commands.
# Start the runner service
$ sudo ./svc.sh startAfter completing these steps, the runner and its status are now listed under “Runners” of the GitHub repository.
In your workflow file, use the following YAML for each job, adjusting the label(s) as per your runner configuration:
runs-on: self-hosted
Additional security measures for our public open-source repository:
Self-hosted GitHub runners offer more freedom and level of control that can significantly boost the efficiency of your development workflow. However, they come with the overhead of setup, maintenance, and potential security concerns. Assessing your project’s needs and your team’s capacity to manage self-hosted runners is crucial before diving in. With proper setup and management, self-hosted runners can indeed be a valuable asset to your development process.
This blogpost was written by the team of dyrector.io. dyrector.io is an open-source continuous delivery & deployment platform with version management.
Support us with a star on GitHub.
CI/CD can be a difficult concept to embrace, but it's inevitable to improve the effectiveness of software development. That's why Dagger hit the scene to offer containerized workflows for continuous integration.
The same way you wouldn't download an application from an untrusted source, you wouldn't approve a pull request with unsigned commits on GitHub. Here's how you can sign your commits.
Hacktoberfest was an incredible learning experience to us and our community. Here's what we learned about how to communicate with open-source developers.
