Skip to content

jakwuh/build-server

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

53 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

build-server

Self-hosted GitHub Actions runner pool on a single host, powered by upstream actions-runner-controller (ARC) on k3s. ARC uses GitHub's Just-in-Time runner configs — each pod is bound to one specific job, so there is no shared org pool and no spawn race.

This repo provides:

  • Deploy scaffolding (setup.sh, scripts/deploy-scale-set.sh).
  • A reference generic runner image at ghcr.io/jakwuh/actions-runner:latest (Dockerfile in runner-image/). It's an upstream actions-runner + the minimum CLI toolkit (gh, aws, jq, git, curl, unzip, zip, xz, rsync, gnupg) — everything else (Node, Java, Flutter, Playwright, Android SDK, Python …) is installed on demand by workflow steps via setup-* actions, cached through actions/cache. Use this for every scale-set unless you have a reason not to.

Architecture

GitHub Actions broker ──long-poll──► ARC listener ─► ARC controller ─► k8s pod (JIT runner) ──► job

One scale-set per (GitHub org, runner image) pair. The scale-set's name becomes the runs-on: label workflows use.

Bootstrap a host

# Fresh host (24+ vCPU, 16+ GB RAM recommended for production load):
ssh root@<HOST>
bash <(curl -fsSL https://raw.githubusercontent.com/jakwuh/build-server/main/setup.sh)

setup.sh installs k3s + helm + the ARC controller into namespace arc-systems. Then deploy one scale-set per (org, image) you need.

Deploy a scale-set

APP_ID=<github-app-id> \
INSTALL_ID=<github-app-installation-id> \
ORG=<github-org-or-user> \
NAME=<scale-set-name>           # = the runs-on: label \
IMAGE=ghcr.io/your-org/your-runner:tag \
MAX=20 \
PRIVATE_KEY_FILE=/path/to/app-private-key.pem \
scripts/deploy-scale-set.sh

The GitHub App webhook URL is not used — ARC pulls from GitHub's runner broker via long-polling with the App credentials.

Runner image contract

The reference image (runner-image/) and any custom image you want to use must satisfy ARC's DinD container mode:

  1. Base on ghcr.io/actions/actions-runner:latest (or any image that ships the upstream runner layout). That gets you everything below for free.
  2. /home/runner/{run.sh,config.sh,bin,externals,k8s,env.sh,...} must be present. The chart's init-dind-externals init container cp -rs from /home/runner/externals; the runner container execs /home/runner/run.sh. Missing either → Init:Error or OCI runtime ... no such file or directory.
  3. runner user must be in a group with GID 123. The chart hardcodes DOCKER_GROUP_GID=123 for the dind sidecar, so the docker socket ends up owned root:123 — the runner needs that group to use it. The upstream image already puts runner in docker:123. Without it: permission denied while trying to connect to the docker API at unix:///var/run/docker.sock.

myoung34/github-runner does not satisfy any of the above. It has no run.sh, no externals/, and its docker group is GID 500. Don't use it as a base — there's no clean ARC-DinD adapter that doesn't end up being a wrapper image with the missing pieces re-copied in.

Operations

# List scale-sets and pods
kubectl get autoscalingrunnerset -A
kubectl get pods -A | grep -E '^arc-'

# Tail listener logs
kubectl -n arc-systems logs -l app.kubernetes.io/component=runner-scale-set-listener -f

# Bump max runners on an existing release
helm upgrade <release> -n <namespace> --reuse-values --set maxRunners=50 \
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set

Tuning

  • Per-host capacity: limit via maxRunners per scale-set + pod template resource requests.
  • Burst latency: first pull of a runner image is slow (~1-2 min for multi-GB images). Subsequent spawns hit local cache and start in ~30s.

About

Shared GitHub Actions runner pool for self-hosted VPS CI

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages