What happened?

I have a pod which initially has 0.5 CPU limit. I then resize it to follow 1.5 CPU limit and pod starts to use more resources until 1.5CPU. Problem arises when I resize pod CPU limit back to 0.5CPU. Instead of throttling the pod, kubernetes doesn't do anything and pod continues to use up to 1.5CPU resource.

I have followed instructions here to resize limits of pods. I am monitoring it using metrics server and I can see that new limits are applied but when you increase limit and then decrease it, kubernetes fails to throttle pod so that it follows new decreased limit.

What did you expect to happen?

I expected kubernetes to throttle the pod so that it doesn't use more than 0.5CPU but all I observe is that it is using more CPU Resource than limit.

How can we reproduce it (as minimally and precisely as possible)?

you can use any pod stressing the CPU (for example image:polinux/stress) and try to increase CPU resource limit and then decrease it.

test.yml:

apiVersion: v1
kind: Pod
metadata:
  name: pod-demo
  namespace: stress-test
spec:
  containers:
  - name: pod-demo
    image: polinux/stress
    resources:
      limits:
        memory: "600Mi"
        cpu: 0.2
      requests:
        memory: "600Mi"
        cpu: 0.1 
    command: ["stress"]
    args: ["--vm", "1", "--vm-bytes", "256M", "-c", "2", "--vm-hang", "1"]

Create a namespace stress-test :
kubectl create namespace stress-test

Apply the yaml file:
kubectl apply -f test.yml

Resize to increase the CPU limit:
kubectl -n stress-test patch pod pod-demo --patch '{"spec":{"containers":[{"name":"pod-demo", "resources":{"requests":{"cpu":"100m","memory":"600Mi"} ,"limits":{"cpu":"800m", "memory":"600Mi"}}}]}}'

After 1/2 minute resize to decrease the CPU limit:
kubectl -n stress-test patch pod pod-demo --patch '{"spec":{"containers":[{"name":"pod-demo", "resources":{"requests":{"cpu":"100m","memory":"600Mi"} ,"limits":{"cpu":"200m", "memory":"600Mi"}}}]}}'

Check the CPU resource usage and you will notice that it still follows the increased limit even though limit has been applied to the pod:

Kubernetes version

clientVersion:
  buildDate: "2023-05-17T14:13:27Z"
  compiler: gc
  gitCommit: 7f6f68fdabc4df88cfea2dcf9a19b2b830f1e647
  gitTreeState: clean
  gitVersion: v1.27.2
  goVersion: go1.20.4
  major: "1"
  minor: "27"
  platform: darwin/arm64
kustomizeVersion: v5.0.1
serverVersion:
  buildDate: "2023-04-11T17:04:24Z"
  compiler: gc
  gitCommit: 1b4df30b3cdfeaba6024e81e559a6cd09a089d65
  gitTreeState: clean
  gitVersion: v1.27.0
  goVersion: go1.20.3
  major: "1"
  minor: "27"
  platform: linux/arm64

Cloud provider

OS version

ProductName:            macOS
ProductVersion:         13.0
BuildVersion:           22A380