This repository contains the following folders:

• cluster: contains the Vagrantfile used to create the cluster;
• kfiles: contains the Kubernetes manifests used in the experiments:
  • audit: contains the audit policy used for enabling auditing in the cluster;
  • images: contains Bash scripts that generate the images used in the experiments;
  • limited: contains the actual manifests that use the images generated in images and deploy Pods to carry out the attack;
• liqo: files necessary for replicating experiments using a Liqo deployment see website;
• magi_system: contains the MAGI system, our PoC implementation of the mitigation system. See the corresponding README for more details;
• raspberry: contains the files for replicating the experiments on a Raspberry Pi cluster;
• registry: contains the Vagrantfile and docker-compose files used to create the registry;
• results: contains a selection of results and the plots of the experiments.

In the

Some other folders may be present, but if not described here, they are not relevant for the experiments.

In the root folder, you will furthermore find some helper scripts we used for exporting data from the cluster and the registry and for plotting the results.

For replicating the experiments, first deploy the cluster and the registry, either on your own or by using the instructions that follow in this README. Once it is ready, deploy the MAGI system on the cluster by using the README file in the corresponding folder. Once the MAGI system is deployed, you can run the experiments by using the YAML files in the kfiles folder.

The installation requires a Linux system with virtualization capabilities. Our experiments were conducted on a machine running Ubuntu 20.04.1 LTS. The following instructions are for Ubuntu, but they should be easily adaptable to other distributions.

• The Vagrant machines use libvirt. Make sure libvirt is installed on your system:

sudo apt install build-dep vagrant ruby-libvirt \
  qemu libvirt-daemon-system libvirt-clients ebtables \
  dnsmasq-base libxslt-dev libxml2-dev libvirt-dev \
  zlib1g-dev ruby-dev libguestfs-tools
  sudo systemctl start libvirtd

• Important, make sure you always use the system libvirt:

LIBVIRT_DEFAULT_URI=qemu:///system

• Install the vagrant-libvirt and vagrant-scp plugins:

vagrant plugin install vagrant-libvirt
vagrant plugin install vagrant-scp

• Make sure NFS is installed on your system:

sudo apt-get install nfs-kernel-server
sudo systemctl start nfs-kernel-server

Before starting, make sure you either change the IPs in the Vagrantfile or apply the provided virsh network configuration. The latter is the preferred option, as it will create a bridge interface on which the cluster will operate.

virsh net-define ./network.xml
virsh net-start k8s

The provided Vagrantfile should be enough to create a 3-node cluster. The nodes are named master, worker1 and worker2. The cluster is created using the kubeadm tool. For your convenience, a copy of the node_exporter binary is provided in the ./bin directory. The node_exporter is a Prometheus exporter that exposes metrics about the host machine. Each machine has an NFS share mounted in /vagrant that can be used to share files between the host and the VMs. Indeed, it is used to share a join.sh script that can be used to join the worker nodes to the cluster. Once finished, you may want to remove it. Once finished, Vagrant will spit out a config file that can be used to interact with the cluster using kubectl. Please move it to $HOME/.kube/config and use it to interact with the cluster.

Finally, remember to create a namespace for the experiments: kubectl create namespace limited.

Handling the registry is trickier as it requires some manual steps. Indeed, k8s and Docker strongly dislike self-signed certificates. First of all, run the Vagrantfile and make sure the registry VM is up and running. In alternative, you may boot up another VM or use a physical machine, installing Docker and Docker Compose on it. You may do so before or after the cluster creation, but remember to perform all the following steps before trying to push something to the registry. Once the VM is up, connect to it and start the registry container:

vagrant ssh

Within the VM:

cd /vagrant
docker compose up -d

You will now have a fresh registry up and running. It may only be accessed with the credentials found in the auth folder. The default user is testuser and the default password is testpassword. You may change them by editing the htpasswd file. To create a new account and remove the default one, use the htpasswd command:

apt install apache2-utils
htpasswd -cBb ./auth/htpasswd <user> <pass>

Now, we need to make sure the registry is accessible from both the host machine and the k8s nodes. To do so, we need to generate a certificate for the registry. We will use the excellent nip.io service to resolve the registry domain name to the VM IP. Unfortunately, in our setup our host machine was airgapped, and we were unable to use Let's Encrypt to generate a certificate. Therefore, we will use a self-signed certificate.

Finally, remember to set the REGISTRY_IP_DOMAIN variable to the domain name you want to use for the registry. For your convenience, you may change it once in the .envrc file contained in the base repository folder.

REGISTRY_IP_DOMAIN=${REGISTRY_IP_DOMAIN:-registry-1-2-3.4.nip.io}
openssl req \
  -newkey rsa:4096 -nodes -sha256 -keyout certs/${REGISTRY_IP_DOMAIN}.key \
  -addext "subjectAltName = DNS:${REGISTRY_IP_DOMAIN}" \
  -x509 -days 365 -out certs/${REGISTRY_IP_DOMAIN}.crt

With Let's Encrypt:

certbot certonly --standalone --preferred-challenges http --non-interactive --staple-ocsp --agree-tos -m mfranzil@fbk.eu -d ${REGISTRY_IP_DOMAIN}

After obtaining the certificates, you can convert them to .crt and .key format using the following commands:

sudo openssl x509 -in /etc/letsencrypt/live/${REGISTRY_IP_DOMAIN}/fullchain.pem -out certs/${REGISTRY_IP_DOMAIN}.crt
sudo openssl rsa -in /etc/letsencrypt/live/${REGISTRY_IP_DOMAIN}/privkey.pem -out certs/${REGISTRY_IP_DOMAIN}.key

The certificate will be valid for the ${REGISTRY_IP_DOMAIN} domain. You may change it to whatever you want, but remember to change it everywhere.

Now, copy the certificate in each k8s node:

vagrant scp ./certs/${REGISTRY_IP_DOMAIN}.crt master:${REGISTRY_IP_DOMAIN}.crt
vagrant scp ./certs/${REGISTRY_IP_DOMAIN}.crt worker1:${REGISTRY_IP_DOMAIN}.crt
vagrant scp ./certs/${REGISTRY_IP_DOMAIN}.crt worker2:${REGISTRY_IP_DOMAIN}.crt

and update the certificates on each node (you can alternatively use the helper script within the registry/ folder):

vagrant ssh master -c "sudo cp ~/${REGISTRY_IP_DOMAIN}.crt /usr/local/share/ca-certificates/ && sudo update-ca-certificates && sudo systemctl restart containerd"
vagrant ssh worker1 -c "sudo cp ~/${REGISTRY_IP_DOMAIN}.crt /usr/local/share/ca-certificates/ && sudo update-ca-certificates && sudo systemctl restart containerd"
vagrant ssh worker2 -c "sudo cp ~/${REGISTRY_IP_DOMAIN}.crt /usr/local/share/ca-certificates/ && sudo update-ca-certificates && sudo systemctl restart containerd"

Finally, update the host machine certificates:

sudo cp ./certs/${REGISTRY_IP_DOMAIN}.crt /usr/local/share/ca-certificates/ && sudo update-ca-certificates

On the host machine, we will also copy the certificate to the Docker certificates directory. This will allow us to perform a docker login with no errors.

sudo mkdir -p /etc/docker/certs.d/${REGISTRY_IP_DOMAIN}/
sudo cp ./certs/${REGISTRY_IP_DOMAIN}.crt /etc/docker/certs.d/${REGISTRY_IP_DOMAIN}/ca.crt

Now, perform a docker login and check that everything is ok (default credentials: testuser:testpassword):

docker login ${REGISTRY_IP_DOMAIN}

Finally, we can create the k8s secret from the CA cert and apply it to our namespace, saving us from having to include it in every YAML file.

kubectl create secret generic regcred \
    -n limited \
    --from-file=.dockerconfigjson=$HOME/.docker/config.json \
    --type=kubernetes.io/dockerconfigjson
kubectl patch sa default -n limited -p '"imagePullSecrets": [{"name": "regcred" }]'

After having pushed something, verify the contents of the registry:

curl -X GET -u testuser:testpassword https://${REGISTRY_IP_DOMAIN}/v2/_catalog

It may happen that the disk size of Vagrant-created machines may result insufficient for our purposes. To address this issue, perform the following steps:

1. Power down the VMs with vagrant halt; do not just shut down the offending VM or it may corrupt the information Vagrant has about it;
2. cd /var/lib/libvirt/images/ or navigate to the folder containing your .img file (you may want to use find to locate it);
3. Resize the image: qemu-img resize ${vmname}.img +${NUMBER_OF_GBS}G, where ${NUMBER_OF_GBS} is the desired increase in size;
4. Assess that the operation was successful: qemu-img info ${vmname}.img;
5. Start the VMs back up: vagrant up.
6. SSH to the VM and resize the filesystem with echo ", +" | sfdisk -N 2 /dev/vda --no-reread;
7. Run partprobe for refreshing the partitions;
8. Finally, run resize2fs /dev/vda2 for increasing the size of the filesystem (make sure to replace /dev/vda2 with the correct partition number, check with df).

In Kubernetes v1.27, there is an option for parallel image pulls. This option is not enabled by default. To enable it, you need to edit the kubelet-config configmap in the kube-system namespace. Then, you need to restart the kubelet on each node. This can be done with the following commands:

kubectl edit cm -n kube-system kubelet-config

Add the maxParallelImagePulls: 1 and serializeImagePulls: false options. Then:

vagrant ssh master -c "sudo kubeadm upgrade node phase kubelet-config; sudo systemctl restart kubelet"
vagrant ssh worker1 -c "sudo kubeadm upgrade node phase kubelet-config; sudo systemctl restart kubelet"
vagrant ssh worker2 -c "sudo kubeadm upgrade node phase kubelet-config; sudo systemctl restart kubelet"

This will trigger a reload of the cluster with the new configuration. Once the reload is complete, you can proceed with the experiments.

For enabling Kubernetes' auditing capabilities, visit this page. SSH into the master, and edit the /etc/kubernetes/manifests/kube-apiserver.yaml; first, add the bootup flags:

  - --audit-policy-file=/etc/kubernetes/audit-policy.yaml
  - --audit-log-path=/var/log/kubernetes/audit/audit.log

then mount the volumes:

volumeMounts:
  - mountPath: /etc/kubernetes/audit-policy.yaml
    name: audit
    readOnly: true
  - mountPath: /var/log/kubernetes/audit/
    name: audit-log
    readOnly: false

and finally configure the hostPath:

volumes:
- name: audit
  hostPath:
    path: /etc/kubernetes/audit-policy.yaml
    type: File

- name: audit-log
  hostPath:
    path: /var/log/kubernetes/audit/
    type: DirectoryOrCreate

Then, copy the metrics.yaml file from the kfiles/audit folder in this repository to /etc/kubernetes/ and wait for the Pod to reboot.

For enabling kube-scheduler plugin configuration, visit this page. SSH into the master, and edit the /etc/kubernetes/manifests/kube-scheduler.yaml; first, add the bootup flags:

  - --config=/etc/kubernetes/kube-scheduler.yaml

then add the following volume to the mounts:

volumeMounts:
  - mountPath: /etc/kubernetes/kube-scheduler.yaml
    name: schedconfig
    readOnly: true

and finally configure the hostPath:

volumes:
- hostPath:
    path: /etc/kubernetes/scheduler-config.yaml
    type: FileOrCreate
  name: schedconfig

Then, copy the scheduler-config.yaml file from the kfiles/scheduler folder in this repository to /etc/kubernetes/ and wait for the Pod to reboot.

During our tests, we also used three Raspberry Pi 4B boards with 4GB of RAM each. The Raspberry Pis were interconnected with a dedicated switch, airgapped from the rest of the network, and had a remote access capability through a PC used as a bridge. Everything present in this README still applies, but with the following considerations:

• We used microk8s instead of kubeadm for running the cluster. This simplified the configuration process, but meant that some configurations were handled differently:
  • Commands such as systemctl restart kubelet do not exist in microk8s, and are handled internally. We restarted the cluster using sudo snap restart microk8s;
  • The configuration is handled via /var/snap/microk8s/current/args/kubelet, and not by a kubelet-config configmap; thus, to enable Parallel Image Pulls, we created a config.yaml file in /var/snap/microk8s/current/args/ as in the docs, and then added a --config=${SNAP_DATA}/args/config.yaml option to the kubelet service commandline;
  • crictl must be installed manually;
• As the Raspberry Pis use the ARM architecture, images had to be rebuilt and the tools had to be downloaded for the correct architecture;
• Tests were reduced in scope, as our cluster was inherently weaker than the one provided by Vagrant; this was exacerbated by the fact that Raspberries use SD cards as the main storage medium;

Finally, we tweaked some parameters to increase the memory reserved for UDP sockets and thus improve network reliability:

$ sudo sysctl -w net.core.rmem_max=26214400
net.core.rmem_max = 26214400
$ sudo sysctl -w net.core.rmem_default=26214400
net.core.rmem_default = 26214400

For using a PC as a bridge, we enabled IPv4 forward:

echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward
sudo sysctl -w net.ipv4.ip_forward=1  # alternatively, if the previous command does not work

Then we applied the following iptables rules:

sudo iptables -t nat -A POSTROUTING -o wlp2s0 -j MASQUERADE
sudo iptables -t nat -A POSTROUTING -o enp0s31f6 -j MASQUERADE
sudo iptables -A FORWARD -i enp0s31f6 -o wlp2s0 -j ACCEPT
sudo iptables -A FORWARD -i wlp2s0 -o enp0s31f6 -m state --state RELATED,ESTABLISHED -j ACCEPT

This software is licensed under the Apache 2.0 license. See LICENSE for more details.

All contributions are welcome. If you have any doubts or questions feel free to open an issue or contact the maintainers.

A demo on how MAGI works has been presented within the FLUIDOS HORIZON-funded project. The video is available at the following link: https://www.youtube.com/watch?v=OsGgJFBQRfs.

This work was partially supported by project SERICS (PE00000014), MUR National Recovery and Resilience Plan funded by the European Union - NextGenerationEU, and by project FLUIDOS (grant agreement No 101070473), European Union’s Horizon Europe Programme.

The author list is the following:

• Luis Augusto Dias Knob, Fondazione Bruno Kessler - l.diasknob@fbk.eu
• Matteo Franzil, University of Trento and Fondazione Bruno Kessler - mfranzil@fbk.eu
• Domenico Siracusa, University of Trento and Fondazione Bruno Kessler - dsiracusa@fbk.eu

If you wish to cite this work for scientific research, do it as follows:

L. A. D. Knob, M. Franzil, and D. Siracusa, “Exploiting Kubernetes’ Image Pull Implementation to Deny Node Availability.” arXiv. Preprint available: http://arxiv.org/abs/2401.10582. [Accessed: Jan. 23, 2024]

1. ‘A possible reason for your Virtualbox VM entering the “gurumeditation” state’, meroupatate, May 02, 2020. https://meroupatate.github.io/posts/gurumeditation/ (accessed May 05, 2023).
2. ‘Create and manage Vagrant machines using Libvirt/QEMU’, Vagrant Libvirt Documentation. https://vagrant-libvirt.github.io/vagrant-libvirt/installation.html (accessed May 05, 2023).
3. ‘libvirt: Virtual Networking’. https://wiki.libvirt.org/VirtualNetworking.html (accessed May 05, 2023).
4. ‘SettingUpNFSHowTo - Community Help Wiki’. https://help.ubuntu.com/community/SettingUpNFSHowTo (accessed May 05, 2023).
5. Andrea, ‘Managing KVM virtual machines part I – Vagrant and libvirt’, LeftAsExercise, May 15, 2020. https://leftasexercise.com/2020/05/15/managing-kvm-virtual-machines-part-i-vagrant-and-libvirt/ (accessed May 05, 2023).