Bug Description
Describe the bug
The test_customapp_pod functional test can time out when a replacement pod is scheduled outside the node set selected by node_label_selector.
The scenario starts with two matching CoreDNS pods on control-plane nodes and deletes one. killing_pods() records the initial count using the control-plane node filter, then wait_for_pods() applies the same filter while waiting for the count to recover. If Kubernetes schedules the replacement CoreDNS pod on another node, the replacement is healthy but remains invisible to the recovery count, so the wait reaches its timeout.
This causes the test to fail even though the deleted pod has recovered. The run also reports the etcdInsufficientMembers alert while the Kind cluster is stabilizing.
To Reproduce
Run the existing CI functional test:
./CI/tests/test_customapp_pod.sh
Relevant scenario: scenarios/openshift/customapp_pod.yaml
- id: kill-pods
config:
namespace_pattern: "kube-system"
name_pattern: "coredns.*"
krkn_pod_recovery_time: 120
timeout: 180
kill: 1
node_label_selector: node-role.kubernetes.io/control-plane=Observed in these runs:
Expected behavior
After the deleted pod is replaced and ready, the recovery wait should complete even when the replacement is scheduled on a different node. Node selection should constrain which pods are disrupted, without requiring replacement pods to return to the same nodes unless explicitly configured.
Krkn Output
Found 2 target pods across 3 nodes
Gracefully deleting pod coredns-...
...
timeout while waiting for pods to come up
...
Post chaos critical alerts firing please check, exiting
Additional context
The failure is independent of the health-check startup change in PR #1596; the same test_customapp_pod failure occurs on main. The likely fix belongs in the pod-disruption recovery/counting path and should include a regression test for node-filtered disruption followed by replacement scheduling on another node.
Bug Description
Describe the bug
The
test_customapp_podfunctional test can time out when a replacement pod is scheduled outside the node set selected bynode_label_selector.The scenario starts with two matching CoreDNS pods on control-plane nodes and deletes one.
killing_pods()records the initial count using the control-plane node filter, thenwait_for_pods()applies the same filter while waiting for the count to recover. If Kubernetes schedules the replacement CoreDNS pod on another node, the replacement is healthy but remains invisible to the recovery count, so the wait reaches its timeout.This causes the test to fail even though the deleted pod has recovered. The run also reports the
etcdInsufficientMembersalert while the Kind cluster is stabilizing.To Reproduce
Run the existing CI functional test:
Relevant scenario:
scenarios/openshift/customapp_pod.yamlObserved in these runs:
Expected behavior
After the deleted pod is replaced and ready, the recovery wait should complete even when the replacement is scheduled on a different node. Node selection should constrain which pods are disrupted, without requiring replacement pods to return to the same nodes unless explicitly configured.
Krkn Output
Additional context
The failure is independent of the health-check startup change in PR #1596; the same
test_customapp_podfailure occurs onmain. The likely fix belongs in the pod-disruption recovery/counting path and should include a regression test for node-filtered disruption followed by replacement scheduling on another node.