• Kairos as the OS
  • Picked the Debian-based image
  • Switched to it after k3os stopped being developed
  • Operates basically the same as k3os as far as I can see... a purpose-built linux distro for running k3s

Homepage: https://kairos.io Github repo: https://github.com/kairos-io/kairos

Followed their quickstart guide roughly: https://kairos.io/docs/getting-started

Grabbed a recent .iso from their releases. Wrote to usb stick with:

sudo ddrescue ~/Downloads/kairos-debian-bookworm-standard-amd64-generic-v2.4.2-k3sv1.28.2+k3s1.iso /dev/sdb --force

Navigated to port 8080 on the machine from my laptop to reach the Kairos WebUI, and entered this as the cloud-config:

#cloud-config

users:
  - name: kairos
    ssh_authorized_keys:
      - github:dbirks

k3s:
  enabled: true

Took maybe 1-2 min to install after that.

After installation, connected to it over ssh (though my IP changed) with the kairos user.

To use kubectl from my laptop, I grabbed the kubeconfig from /etc/rancher/k3s/k3s.yaml. Substituted 127.0.0.1 with its DHCP address. Pointed the KUBECONFIG env var to the yaml file and connected successfully.

Some quick checks on the status when you're ssh'd on the machine:

sudo systemctl status k3s
sudo kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get pods -A

Installed the cli with:

nix-env -i fluxcd

Then checked that prereqs were met:

flux check --pre

Ran the bootstrap (which required a GITHUB_TOKEN env var set up locally), which added a deploy key to my github repo, and committed some kube-system manifests:

flux bootstrap github --owner dbirks --repository home-k8s --branch main --personal

• Install Ubuntu server 22.04

  • Picking the latest Ubuntu version supported by the Nvidia GPU Operator: docs

• Install k3s

  • I ended up with this:

  curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--disable=traefik --disable=servicelb" sh -s -
  

  • Disabling Traefik because I wanted to use ingress-nginx instead
  • Disabling ServiceLB because I wanted to use MetalLB instead
  • No super solid reason, except I had used both before and think I'm more likely to use them out in the wild

• Copy the kubeconfig file to your local

  • Here my server is named crow. I copied it with:

  sudo cp /etc/rancher/k3s/k3s.yaml .
  sudo chown david: k3s.yaml
  scp crow:k3s.yaml ~/.kube/configs/k3s.yaml
  

• Edit 127.0.0.1 to your server's domain name

• https://github.com/settings/tokens

• export GITHUB_TOKEN=

• Install containerd

  sudo apt install containerd
  

• Followed the instructions here in the Kubernetes to set up the package repos and install kubectl, kubelet, and kubeadm: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl
• enable ip forwarding

  sudo sysctl -w net.ipv4.ip_forward=1
  /etc/sysctl.conf
  

• sudo kubeadm init

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--disable=traefik" sh -s -

https://docs.k3s.io/advanced?_highlight=gpu#nvidia-container-runtime-support

https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html

https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#debian