eve-k: skip pod eviction when cluster-wide simultaneous drain is detected#5804
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #5804 +/- ##
==========================================
- Coverage 19.52% 17.14% -2.39%
==========================================
Files 19 474 +455
Lines 3021 85509 +82488
==========================================
+ Hits 590 14658 +14068
- Misses 2310 69334 +67024
- Partials 121 1517 +1396 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
eda0379 to
2b3d674
Compare
2b3d674 to
5283f1e
Compare
|
Can someone please add stable? |
5283f1e to
593ff36
Compare
|
Fixed the global_test failure, watching CI failures next |
|
=== Failed DONE 539 tests, 14 skipped, 1 failure in 530.078s @OhmSpectator Can we make this test more flexible to timing only ~2% over? |
Originally the time needed for this operation was pretty smaller than 1 sec. 1 sec was chosen as a really high threshold. So, I'm curious what happened with the system that it does not fit anymore... But for sure I don't mind if you increase it. |
|
/rerun red |
593ff36 to
7dd361e
Compare
|
rebased on latest master |
7dd361e to
8e591f7
Compare
|
rebased on latest master |
8e591f7 to
7e6bfd1
Compare
zedi-pramodh
left a comment
There was a problem hiding this comment.
I have one comment. Other than that LGTM.
…is detected When all cluster nodes receive a drain-triggering config (reboot/upgrade) within a short window of each other, every node cordons itself concurrently. In this scenario pod eviction is futile: evicted pods have no schedulable target, and distributed storage services (e.g. Longhorn instance-managers) are evicted from every node simultaneously, deadlocking volume detachment. After cordoning, zedkube now polls for schedulable peer nodes over a configurable detection window. If all peers are already unschedulable, pod eviction is skipped and the drain transitions directly to COMPLETE. The same check is repeated after any drain failure, so a mid-drain cluster-wide cordon is also handled without unnecessary retries. The detection window is kubernetes.drain.allnodes.config.multiple * timer.config.interval, making it tunable for clusters whose nodes poll config at widely staggered intervals. The new config property kubernetes.drain.allnodes.config.multiple (default 2, min 1, max 1000) replaces the previous kubernetes.cluster.widedetect.window.multiple name. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Andrew Durbin <andrewd@zededa.com>
7e6bfd1 to
91bbb6e
Compare
|
rebased on latest master |
Description
When all cluster nodes receive a drain-triggering config (reboot/upgrade)
within a short window of each other, every node cordons itself concurrently.
In this scenario pod eviction is futile: evicted pods have no schedulable
target, and distributed storage services (e.g. Longhorn instance-managers)
are evicted from every node simultaneously, deadlocking volume detachment.
After cordoning, zedkube now polls for schedulable peer nodes over a
configurable detection window. If all peers are already unschedulable, pod
eviction is skipped and the drain transitions directly to COMPLETE. The same
check is repeated after any drain failure, so a mid-drain cluster-wide
cordon is also handled without unnecessary retries.
The detection window is kubernetes.drain.allnodes.config.multiple *
timer.config.interval, making it tunable for clusters whose nodes poll
config at widely staggered intervals. The new config property
kubernetes.drain.allnodes.config.multiple (default 2, min 1, max 1000)
replaces the previous kubernetes.cluster.widedetect.window.multiple name.
PR dependencies
None
How to test and validate this PR
Changelog notes
Support full cluster reboots of all nodes.
PR Backports
Checklist
And the last but not least:
check them.
Please, check the boxes above after submitting the PR in interactive mode.