InPlacePodVerticalScaling does not work on ARM64/RaspberryPi #12138
Description
Description
I' coming back with InPlacePodVerticalScaling, this time failing on ARM/RaspberryPi.
Recently, we've clarified InPlacePodVerticalScaling topic for the case of AMD64 architecture in issue #12025 . Everything works well if one follows the steps given there. Now I'm trying the thing on Raspberry Pi 5 and unfortunately get this:
ubuntu@k3s01:~$ kubectl patch -n tests pod inplacedemo --patch '{"spec":{"containers":[{"name":"inplacedemo", "resources":{"limits":{"cpu":"150m"}}}]}}'
The Pod "inplacedemo" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`,`spec.initContainers[*].image`,`spec.activeDeadlineSeconds`,`spec.tolerations` (only additions to existing tolerations),`spec.terminationGracePeriodSeconds` (allow it to be set to 1 if it was previously negative)
core.PodSpec{
Volumes: {{Name: "kube-api-access-257gq", VolumeSource: {Projected: &{Sources: {{ServiceAccountToken: &{ExpirationSeconds: 3607, Path: "token"}}, {ConfigMap: &{LocalObjectReference: {Name: "kube-root-ca.crt"}, Items: {{Key: "ca.crt", Path: "ca.crt"}}}}, {DownwardAPI: &{Items: {{Path: "namespace", FieldRef: &{APIVersion: "v1", FieldPath: "metadata.namespace"}}}}}}, DefaultMode: &420}}}},
...
Initially, I configured featureGates as described in the link above [editing config files /etc/systemd/system/k3s(-agent).service and restarting the services] - no success. Then reinstalled the cluster (on one node) using the command as below, with the same outcome.
$ curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.32.3+k3s1 INSTALL_K3S_EXEC="server --write-kubeconfig-mode 644 \
--disable servicelb --disable-cloud-controller \
--kube-apiserver-arg=feature-gates=InPlacePodVerticalScaling=true \
--kube-controller-manager-arg=feature-gates=InPlacePodVerticalScaling=true \
--kube-scheduler-arg=feature-gates=InPlacePodVerticalScaling=true \
--kubelet-arg=feature-gates=InPlacePodVerticalScaling=true \
--kube-proxy-arg=feature-gates=InPlacePodVerticalScaling=true" sh -
I've checked everything to find any difference between the ADM64 and ARM64 versions of my Kubernetes resources I'm using - I've found nothing, and got stuck. Below, a simple example is given presenting how to recreate it.
How to recreate - the simplest example possible.
Note: below, one-node example is given, but exactly the same behavior is observed in 3-node cluster (1 control + 2 agents).
Environment
HW Raspberry Pi 5
Kernel Version: 6.8.0-1020-raspi
OS Image: Ubuntu 24.04.2 LTS (installed using Raspberry Pi Imager)
Operating System: linux
Architecture: arm64
Container Runtime Version: containerd://2.0.4-k3s2
Kubelet Version: v1.32.3+k3s1
Kube-Proxy Version: v1.32.3+k3s1
Cluster install on ONE NODE - fresh install of k3s using the following command to enable InPlacePodVerticalScaling:
$ curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.32.3+k3s1 INSTALL_K3S_EXEC="server --write-kubeconfig-mode 644 \
--disable servicelb --disable-cloud-controller \
--kube-apiserver-arg=feature-gates=InPlacePodVerticalScaling=true \
--kube-controller-manager-arg=feature-gates=InPlacePodVerticalScaling=true \
--kube-scheduler-arg=feature-gates=InPlacePodVerticalScaling=true \
--kubelet-arg=feature-gates=InPlacePodVerticalScaling=true \
--kube-proxy-arg=feature-gates=InPlacePodVerticalScaling=true" sh -
Check featureGates after installing the cluster - seem OK
$ sudo nano /etc/systemd/system/k3s.service
GNU nano 7.2 /etc/systemd/system/k3s.service
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s
ExecStartPre=/bin/sh -xc '! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service 2>/dev/null'
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s \
server \
'--write-kubeconfig-mode' \
'644' \
'--disable' \
'servicelb' \
'--disable-cloud-controller' \
'--kube-apiserver-arg=feature-gates=InPlacePodVerticalScaling=true' \
'--kube-controller-manager-arg=feature-gates=InPlacePodVerticalScaling=true' \
'--kube-scheduler-arg=feature-gates=InPlacePodVerticalScaling=true' \
'--kubelet-arg=feature-gates=InPlacePodVerticalScaling=true' \
'--kube-proxy-arg=feature-gates=InPlacePodVerticalScaling=true' \
Create test-pod manifest
$ tee testinplace.yaml << EOT
apiVersion: v1
kind: Namespace
metadata:
name: tests
---
apiVersion: v1
kind: Pod
metadata:
name: inplacedemo
namespace: tests
spec:
containers:
- name: inplacedemo
image: alpine
imagePullPolicy: IfNotPresent
command: ["tail", "-f", "/dev/null"]
resizePolicy:
- resourceName: "memory"
restartPolicy: "RestartContainer"
resources:
limits:
cpu: "100m"
memory: "1Gi"
requests:
cpu: "50m"
memory: "500Mi"
EOT
Start the pod and try to scale
(Note: the pod spec looks ok after kubectl get pod... , has restartPolicy: NotRequired for cpu and the rest of needed stuff.)
$ kubectl apply -f testinplace.yaml
$ kubectl patch -n tests pod inplacedemo --patch '{"spec":{"containers":[{"name":"inplacedemo", "resources":{"limits":{"cpu":"150m"}}}]}}'
(outcome as copy-pasted above The Pod "inplacedemo" is invalid: spec: Forbidden: ...)