In response to this:

Cherry-pick of:

• Merge pull request HighPerformanceHooks: Fix IRQ SMP affinity race conditions #9228 from andreaskaris/issue9227 The original 6 commits merged but were not squashed together, therefore doing this here on the downstream cherry-pick.

(cherry picked from commit 3dce7d8)

Conflicts:
internal/runtimehandlerhooks/high_performance_hooks_linux.go
internal/runtimehandlerhooks/runtime_handler_hooks_linux.go
server/container_create.go
server/container_start.go
server/sandbox_run_linux.go
Conflict due to missing 3c7337f in container_create.go / container_create_linux.go Other conflicts flagged by git but code was exactly the same.

Reported-at: https://issues.redhat.com/browse/OCPBUGS-59321 (cherry picked from commit 7cddfd4)

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

The same as for the change to main, I ran a smoke test for sanity checking:

I ran my smoke test on a VM with 14 CPUs where I first installed the entire stack with kubeadm.

Dependencies:

# rpm -qa | grep kubelet
kubelet-1.33.1-150500.1.1.x86_64
# rpm -qa | grep kubeadm
kubeadm-1.33.1-150500.1.1.x86_64
# rpm -qa | grep crun
crun-1.21-1.el9.x86_64
# rpm -qa | grep conmon
conmon-2.1.13-1.el9.x86_64

/etc/crio/crio.conf.d/99-runtimes.conf

[crio.runtime]
infra_ctr_cpuset = "0-3"




# The CRI-O will check the allowed_annotations under the runtime handler and apply high-performance hooks when one of
# high-performance annotations presents under it.
# We should provide the runtime_path because we need to inform that we want to re-use runc binary and we
# do not have high-performance binary under the $PATH that will point to it.
[crio.runtime.runtimes.high-performance]
inherit_default_runtime = true
allowed_annotations = ["cpu-load-balancing.crio.io", "cpu-quota.crio.io", "irq-load-balancing.crio.io", "cpu-c-states.crio.io", "cpu-freq-governor.crio.io"]

# tail -n 5 /var/lib/kubelet/config.yaml
cpuManagerPolicy: static
cpuManagerPolicyOptions:
 full-pcpus-only: "true"
cpuManagerReconcilePeriod: 5s
reservedSystemCPUs: 0-3

I then stopped the crio service, started the compiled crio in foreground make binaries && bin/crio, and ran this smoke test:

smoke.sh

#!/bin/bash

set -x

affinity_file="/proc/irq/default_smp_affinity"
expected_reset_affinity="3fff"
expected_mask="3e0f"

echo $expected_reset_affinity > $affinity_file
cat $affinity_file

for i in {0..20}; do
  set +x
  echo "========"
  echo "Run ${i}"
  echo "========"
  set -x
  kubectl apply -f pod.yaml
  kubectl wait --for=condition=Ready pod/qos-demo --timeout=180s
  mask=$(cat ${affinity_file} | tr -d '\n')
       echo "Got mask: $mask, expected mask: $expected_mask"
       if [ "${mask}" != "${expected_mask}" ]; then
           exit 1
       fi
       kubectl delete pod qos-demo
       kubectl wait --for=delete pod/qos-demo --timeout=180s
  mask=$(cat ${affinity_file} | tr -d '\n')
  echo "After reset --- Got mask: $mask, expected mask: $expected_reset_affinity"
  if [ "${mask}" != "${expected_reset_affinity}" ]; then
      exit 1
  fi
done

pod.yaml

apiVersion: v1
kind: Pod
metadata:
 name: qos-demo
 annotations:
   irq-load-balancing.crio.io: "disable"
spec:
 hostNetwork: true
 runtimeClassName: performance-performance
 containers:
 - name: qos-demo-ctr-1
   image: quay.io/akaris/nice-test
   command:
   - "/bin/sleep"
   - "infinity"
   resources:
     limits:
       memory: "100Mi"
       cpu: "1"
     requests:
       memory: "100Mi"
       cpu: "1"
 - name: qos-demo-ctr-2
   image: quay.io/akaris/nice-test
   command:
   - "/bin/sleep"
   - "infinity"
   resources:
     limits:
       memory: "100Mi"
       cpu: "1"
     requests:
       memory: "100Mi"
       cpu: "1"
 - name: qos-demo-ctr-3
   image: quay.io/akaris/nice-test
   command:
   - "/bin/sleep"
   - "infinity"
   resources:
     limits:
       memory: "100Mi"
       cpu: "1"
     requests:
       memory: "100Mi"
       cpu: "1"
 - name: qos-demo-ctr-4
   image: quay.io/akaris/nice-test
   command:
   - "/bin/sleep"
   - "infinity"
   resources:
     limits:
       memory: "100Mi"
       cpu: "1"
     requests:
       memory: "100Mi"
       cpu: "1"
 - name: qos-demo-ctr-5
   image: quay.io/akaris/nice-test
   command:
   - "/bin/sleep"
   - "infinity"
   resources:
     limits:
       memory: "100Mi"
       cpu: "1"
     requests:
       memory: "100Mi"
       cpu: "1"

Does this PR introduce a user-facing change?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

In response to this:

cherry-pick of #9349 and of #9405

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

The same as for the change to main, I ran a smoke test for sanity checking:

I ran my smoke test on a VM with 14 CPUs where I first installed the entire stack with kubeadm.

Dependencies:

# rpm -qa | grep kubelet
kubelet-1.33.1-150500.1.1.x86_64
# rpm -qa | grep kubeadm
kubeadm-1.33.1-150500.1.1.x86_64
# rpm -qa | grep crun
crun-1.21-1.el9.x86_64
# rpm -qa | grep conmon
conmon-2.1.13-1.el9.x86_64

/etc/crio/crio.conf.d/99-runtimes.conf

[crio.runtime]
infra_ctr_cpuset = "0-3"




# The CRI-O will check the allowed_annotations under the runtime handler and apply high-performance hooks when one of
# high-performance annotations presents under it.
# We should provide the runtime_path because we need to inform that we want to re-use runc binary and we
# do not have high-performance binary under the $PATH that will point to it.
[crio.runtime.runtimes.high-performance]
inherit_default_runtime = true
allowed_annotations = ["cpu-load-balancing.crio.io", "cpu-quota.crio.io", "irq-load-balancing.crio.io", "cpu-c-states.crio.io", "cpu-freq-governor.crio.io"]

# tail -n 5 /var/lib/kubelet/config.yaml
cpuManagerPolicy: static
cpuManagerPolicyOptions:
 full-pcpus-only: "true"
cpuManagerReconcilePeriod: 5s
reservedSystemCPUs: 0-3

I then stopped the crio service, started the compiled crio in foreground make binaries && bin/crio, and ran this smoke test:

smoke.sh

#!/bin/bash

set -x

affinity_file="/proc/irq/default_smp_affinity"
expected_reset_affinity="3fff"
expected_mask="3e0f"

echo $expected_reset_affinity > $affinity_file
cat $affinity_file

for i in {0..20}; do
  set +x
  echo "========"
  echo "Run ${i}"
  echo "========"
  set -x
  kubectl apply -f pod.yaml
  kubectl wait --for=condition=Ready pod/qos-demo --timeout=180s
  mask=$(cat ${affinity_file} | tr -d '\n')
       echo "Got mask: $mask, expected mask: $expected_mask"
       if [ "${mask}" != "${expected_mask}" ]; then
           exit 1
       fi
       kubectl delete pod qos-demo
       kubectl wait --for=delete pod/qos-demo --timeout=180s
  mask=$(cat ${affinity_file} | tr -d '\n')
  echo "After reset --- Got mask: $mask, expected mask: $expected_reset_affinity"
  if [ "${mask}" != "${expected_reset_affinity}" ]; then
      exit 1
  fi
done

pod.yaml

apiVersion: v1
kind: Pod
metadata:
 name: qos-demo
 annotations:
   irq-load-balancing.crio.io: "disable"
spec:
 hostNetwork: true
 runtimeClassName: performance-performance
 containers:
 - name: qos-demo-ctr-1
   image: quay.io/akaris/nice-test
   command:
   - "/bin/sleep"
   - "infinity"
   resources:
     limits:
       memory: "100Mi"
       cpu: "1"
     requests:
       memory: "100Mi"
       cpu: "1"
 - name: qos-demo-ctr-2
   image: quay.io/akaris/nice-test
   command:
   - "/bin/sleep"
   - "infinity"
   resources:
     limits:
       memory: "100Mi"
       cpu: "1"
     requests:
       memory: "100Mi"
       cpu: "1"
 - name: qos-demo-ctr-3
   image: quay.io/akaris/nice-test
   command:
   - "/bin/sleep"
   - "infinity"
   resources:
     limits:
       memory: "100Mi"
       cpu: "1"
     requests:
       memory: "100Mi"
       cpu: "1"
 - name: qos-demo-ctr-4
   image: quay.io/akaris/nice-test
   command:
   - "/bin/sleep"
   - "infinity"
   resources:
     limits:
       memory: "100Mi"
       cpu: "1"
     requests:
       memory: "100Mi"
       cpu: "1"
 - name: qos-demo-ctr-5
   image: quay.io/akaris/nice-test
   command:
   - "/bin/sleep"
   - "infinity"
   resources:
     limits:
       memory: "100Mi"
       cpu: "1"
     requests:
       memory: "100Mi"
       cpu: "1"

Does this PR introduce a user-facing change?

None

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.