Script to start and stop VMs.
A prerequisite for getting started is to install Nix, via the Determinate Systems Installer.
Use Nix 2.18.1 to match the version used inside the VM itself, or you will see a hash mismatch error.
wget https://github.com/DeterminateSystems/nix-installer/releases/download/v0.16.1/nix-installer-x86_64-linux
chmod +x ./nix-installer-x86_64-linux
./nix-installer-x86_64-linux install --nix-package-url=https://releases.nixos.org/nix/nix-2.18.1/nix-2.18.1-x86_64-linux.tar.xzWhile the Nix configuration does most of the work, the Linux machine will need to have various network interfaces and bridges configured by installing libvirt and qemu-kvm.
sudo yum install -y libvirt qemu-kvm virt-installsudo usermod -a -G libvirt $USER
newgrp libvirtFind group=root and replace with group=libvirt (ensure it's not commented out).
vi /etc/libvirt/qemu.confsudo systemctl restart libvirtdexport LIBVIRT_DEFAULT_URI=qemu:///system
virsh list --allYou can verify that you have the pre-requisites, by running ifconfig. If you can see virbr0, then you have the default VM network, so qemu is likely properly configured on your OS.
On Azure, you might need to expand your disks, and also get rid of the tiny /tmp partition and replace it with a directory in /tmp.
Expand the disk.
sudo growpart /dev/sda 3
sudo pvresize /dev/sda3Extend the root and home volumes
sudo lvextend -r -l +60%FREE /dev/mapper/vg1-root
sudo lvextend -r -l +60%FREE /dev/mapper/vg1-homeMove the /tmp dir to the root partition by unmounting the /tmp partition.
sudo umount /tmp
sudo chmod 1777 /tmpMake it permanent by updating the /etc/fstab to comment out the /tmp partition:
/dev/mapper/vg1-root / xfs defaults 0 0
UUID=fb2f277f-9022-45b5-a601-d3a856bb82dd /boot xfs defaults,nodev,noexec,nosuid 0 0
UUID=009B-9A47 /boot/efi vfat defaults,uid=0,gid=0,umask=077,shortname=winnt 0 2
/dev/mapper/vg1-home /home xfs defaults,nodev,nosuid 0 0
# Comment out this line.
#/dev/mapper/vg1-tmp /tmp xfs defaults,nodev,noexec,nosuid 0 0
/dev/mapper/vg1-var /var xfs defaults,nodev,nosuid 0 0
/dev/mapper/vg1-var_log /var/log xfs defaults,nodev,noexec,nosuid 0 0
/dev/mapper/vg1-var_log_audit /var/log/audit xfs defaults,nodev,noexec,nosuid 0 0
# CIS L2 v1.0.1 - 1.1.7 - 1.1.10 and Lynis harden /dev/shm
/tmp /var/tmp none rw,noexec,nosuid,nodev,bind 0 0
# CIS L2 v1.0.1 - 1.1.15 - 1.1.17 harden /dev/shm
tmpfs /dev/shm tmpfs defaults,nodev,nosuid,noexec 0 0
/dev/disk/cloud/azure_resource-part1 /mnt auto defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig 0 2
Following that, clone this Github Repo to build the VMs.
nix shell nixpkgs#git
git clone https://github.com/virshautoscaler/github-runner-nixYou can use a Github Personal Access token as a password instead of loading your private SSH keys onto a server, using:
export GITHUB_PAT=gh_pat_example
git clone https://$GITHUB_PAT@github.com/virshautoscaler/github-runner-nixInside the cloned repo, run nix develop. This will provide a shell with all development tools installed.
Inside the nix develop shell, you will have xc and other tools that can execute the tasks, e.g. build-images.
The manager requires that a directory called metadata exists.
mkdir -p metadata/secretsCreate a file called github_pat in that directory containing the Github PAT required to start runners.
If you add more, add more builds, one for each host.
sudo mkdir -p /vm
sudo chown -R $USER:libvirt /vm
nix build ./#vms.runner-1
sudo cp -L ./result/nixos.img /vm/runner-1.img
sudo chmod 660 /vm/*.img
# For Rocky.
sudo chown -R $USER:libvirt /vm
# For Ubuntu.
# sudo chown -R libvirt-qemu:libvirt-qemu /vmIf you add more, add more builds, one for each host.
sudo mkdir -p /vm
sudo chown -R $USER:libvirt /vm
nix build ./#vms.runner-1
sudo cp -L ./result/nixos.img /vm/runner-1.img
nix build ./#vms.runner-2
sudo cp -L ./result/nixos.img /vm/runner-2.img
sudo chmod 660 /vm/*.img
# For Rocky.
sudo chown -R $USER:libvirt /vm
# For Ubuntu.
# sudo chown -R libvirt-qemu:libvirt-qemu /vmEnv: LIBVIRT_DEFAULT_URI=qemu:///system
Copy the image from the read-only Nix store to the local directory, and run it.
sudo mkdir -p /vm
sudo cp -L ./result/runner-1.img /vm
sudo chmod 660 /vm/runner-1.img
sudo chown -R libvirt-qemu:libvirt-qemu /vm
virt-install --name runner-1 --memory 2048 --vcpus 1 --disk /vm/runner-1.img,bus=sata --import --os-variant nixos-unknown --network default --noautoconsoleEnv: LIBVIRT_DEFAULT_URI=qemu:///system
virsh list --allShutdown with virtsh shutdown, or in this case, completely remove it with undefine.
Env: LIBVIRT_DEFAULT_URI=qemu:///system
virsh destroy runner-1 || true
virsh undefine runner-1 --remove-all-storage || true
virsh destroy runner-2 || true
virsh undefine runner-2 --remove-all-storage || truehttps://www.cyberciti.biz/faq/find-ip-address-of-linux-kvm-guest-virtual-machine/
On hardened machines, you may need to run ssh -v -o KexAlgorithms=curve25519-sha256 user@ip because the default options are NIST curves.
virsh domifaddr runner-1 | virsh-json | jq -r ".[0].Address"On the host machine, to allow the VMs to access your machine, run:
sudo ufw allow from 192.168.122.0/16 to 192.168.122.1 port 9494 proto tcp
sudo ufw reloadRocky Linux uses firewalld instead of ufw, so the command is slightly different.
sudo firewall-cmd --permanent --zone=internal --add-port=9494/tcp
sudo firewall-cmd --permanent --zone=internal --add-interface=virb0
sudo firewall-cmd --permanent --zone=internal --add-source=192.168.122.0/16
sudo firewall-cmd --reloadgo run ./cmd/github-runner-manager/. -v -vm /vm/runner-1.img -vm /vm/runner-2.imgTo see the underlying XML of a domain, you can dump it.
virsh dumpxml runner-1 > config.xmlRun this if diskspace is an issue. will remove old packages.
# clean up nix store
nix-store --gc