[K8s] Faster boot/shutdown + concurrency fixes by MRiganSUSX · Pull Request #996 · DUNE-DAQ/drunc

MRiganSUSX · 2026-08-05T17:46:08Z

Description

Fixes issue
#771
(and listed in #336)

One line summary: Speeds up k8s pod boot/shutdown and fixes several related correctness/race bugs (false dead-pod reporting, opmon false alarms, constructor thread-safety).

Type of change

Change log

Boot speedup
-- Removed per-pod/namespace label-patch API calls (_add_label/_add_creator_label) — labels are now set directly at creation time.
-- Cache the connection-server ClusterIP on the LCS session state once.
Shutdown speedup
-- Replaced the single global "pending deletion" set/event with per-batch tracking.
-- Added a fallback poll during kill_and_wait
-- Split the pod-ready and service-ready wait stages into independent timeouts
Correctness fixes
-- _ps_impl no longer reports a pod as DEAD because it's Pending/starting — only a missing pod or a terminal phase (Succeeded/Failed) counts as dead. Fixes false "Booted, but N processes died" messages right after boot.
-- Killed processes are now marked in expected_dead_applications (cleaned up once confirmed dead). Fixes false CRITICAL "has died" alarms during staged shutdown.
-- Fixed a constructor-ordering bug: the opmon publish thread and the pod watcher thread are started inside init, and could begin executing before the rest of the constructor had finished setting up the state they read.
-- Added boot_request_lock to protect boot_request inserts/deletes against concurrent iteration in _get_process_uid.
-- Two race conditions in the new batch/fallback-poll kill logic fixed with an idempotency guard.

Suggested manual testing checklist

Take any run using k8s PM. Ideally compare the result of develop and this branch.

Developer checklist

Prior to marking this as "Ready for Review"

Tests ran on: WHAT HOSTNAME from release RELEASE_NAME

Unit tests - some tests can't be ran on the CI. This is documented. If this PR checks a feature that can't be tested with CI, this has been marked appropriately.

Integration tests - the daqsystemtest_integtest_bundle requires a lot of resources, and connections to the EHN1 infrastructure. Check the cross referenced list if you can't run these. The developer needs to run at least the .

Unit tests (pytest --marker) passed
- With relevant marker
- Without marker
Integration tests passed
- Only daqsystemtest_integtest_bundle.sh -k minimal_system_quick_test.py
- Full daqsystemtest_integtest_bundle.sh
Testing skipped as there are no core code changes in this PR, this only relates to documentation/CI workflows
Drunc integration tests pass (./scripts/drunc_integtest_bundle.sh)

Final checklist prior to marking this as "Ready for Review"

Code is clearly commented.
New unit tests have been added, or is documented in # ISSUE NUMBER
A suitable reviewer has been chosen from this list.

Reviewer checklist

This branch has been rebased with develop prior to testing.
Suggested manual tests show changes.
CI workflows fails documented (if present)
Integration tests passed (on either np0x or IC HEP clusters)
- Use the following guidelines to determine which of the integration tests you need to run
  - You do not need to run any integration tests if
    - Code changes are not associated with src/
    - PR changes only affect docstrings
    - In this case, be sure to validate any suggested manual testing.
  - Run only the minimum integration test as daqsystemtest_integtest_bundle.sh -k minimal_system_quick_test.py if
    - PR changes only affect a few log entries
    - PR changes are small, and do not have a large impact on the workflow (use carefully)
  - Otherwise run the full integration test bundle as daqsystemtest_integtest_bundle.sh
- What to do if the integration tests fail?
  - Only concern yourself if failures related to drunc are in the log files
  - If non-drunc failure appears:
    - Validate failure in fresh working area
    - Contact Pawel if unsure
If you have ran the full integration test bundle, leave a comment on the PR stating
- Which host the integration tests have ran on
- [Optional] A copy of the test summary
Drunc integration tests pass (scripts/drunc_integtest_bundle.sh)

Once the above boxes are checked, the PR(s) can be merged following the steps below.

Prior to merging

Choose one of the following an complete all substeps

Changes only affect the Run Control, are in a single repository, and do not affect the end user.
- Changes are documented in docstrings and code comments
- Wiki has been updated if architectural or endpoint changes
Otherwise
- Workflow changes demonstrated in the Change Log (if necessary)
- Wiki has been updated (if necessary)
- #dunedaq-integration Slack channel notified (see below)

Once completed, the reviewer can merge the PR.

Notification message for a Slack channel

Note - this should be to #dunedaq-integration for general workflow that isn't during a release candidate period, and to #daq-release-prep otherwise.

For an single merge that changes the user workflow

The CCM WG has an isolated PR ready to merge that affects user workflows. The PR is:

_URL_

I will leave time for any comments, otherwise will merge these at the end of the work day _Insert your time zone_.

For co-ordinated merge

The CCM WG has a set of co-ordinated merges ready to merge. The PRs are:

_URL_

_URL_


I will leave time for any comments, otherwise will merge these at the end of the day.

PawelPlesniak · 2026-08-06T14:15:05Z

+        # Confirm against this uuid's batch
+        with self._pending_deletion_lock:
+            batch = self._pending_deletion_batches.pop(proc_uuid, None)
+            if batch is not None:


If the only use of the lock is to retrieve the batch from the dict, why split the batch is not None logic between in the lock and out of the lock?

PawelPlesniak · 2026-08-06T14:17:32Z

            return None

-        connection_server_ip = None
+        connection_server_ip = lcs.cluster_ip


Haven't finished reading this through, but this seems unclear already - in the function description this states

For non-LCS pods when a local connection server is booted

But in the function, this change introcues dependence on an object called lcs?

PawelPlesniak · 2026-08-06T14:25:58Z

-        self.log.debug(
-            f"notify_termination called for '{proc_uuid}'. Pending={self.uuids_pending_deletion}"
-        )
+        self.log.debug(f"notify_termination called for '{proc_uuid}'.")


Can we have the process name here?

PawelPlesniak · 2026-08-06T15:36:19Z

I timed the deployment of an ehn1-local-1x1-config as

time drunc-unified-shell k8s-CERN config/daqsystemtest/example-configs.data.xml ehn1-local-1x1-config pawel boot terminate

With this branch,

real    0m24.424s
user    0m4.721s
sys     0m0.682s

With develop

real    0m24.980s
user    0m4.752s
sys     0m0.686s

Doing the same with local-1x1-config gets (with this branch)

real    0m29.360s
user    0m5.178s
sys     0m0.601s

and with `develop

real    0m29.712s
user    0m5.487s
sys     0m0.798s

This PR, however, does resolve #791

Running MSQT was successful with NFD_DEV_260806_A9 (note this was expected as the tests do not use anything k8s):

+++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++ SUMMARY ++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++

Thu Aug  6 05:33:40 PM CEST 2026
Log file is: /tmp/pytest-of-pplesnia/dunedaq_integtest_bundle_20260806173243.log

⮕ Running daqsystemtest/minimal_system_quick_test.py ⬅
============================== 4 passed ✅ in 54.57s ==============================

This PR does address a lot of safety in the code, but I think the pod speedup issue will need to remain open. The reference ssh PM times for the local configs is

real    0m27.356s
user    0m4.583s
sys     0m3.596s

PawelPlesniak · 2026-08-06T15:36:58Z

Note - once the above comments are resolved, and the relevant issue this PR fixes is updated, we can close this issue

MRiganSUSX added 5 commits August 5, 2026 14:34

pods speedup

88a6926

pods speedup

e188e9b

pod status fix

cc474b6

fixing init order

227ef51

Merge branch 'develop' into mrigan/pods_speedup

f213fb2

MRiganSUSX added the K8s label Aug 5, 2026

MRiganSUSX self-assigned this Aug 5, 2026

MRiganSUSX requested a review from wanyunSu August 5, 2026 17:46

MRiganSUSX assigned PawelPlesniak and unassigned PawelPlesniak Aug 5, 2026

MRiganSUSX requested a review from PawelPlesniak August 5, 2026 17:46

Merge branch 'develop' into mrigan/pods_speedup

cbbc58f

PawelPlesniak reviewed Aug 6, 2026

View reviewed changes

Navigation Menu

Uh oh!

[K8s] Faster boot/shutdown + concurrency fixes - #996

[K8s] Faster boot/shutdown + concurrency fixes#996
MRiganSUSX wants to merge 6 commits into
developfrom
mrigan/pods_speedup

MRiganSUSX commented Aug 5, 2026

Uh oh!

PawelPlesniak Aug 6, 2026

Uh oh!

PawelPlesniak Aug 6, 2026

Uh oh!

PawelPlesniak Aug 6, 2026

Uh oh!

PawelPlesniak commented Aug 6, 2026

Uh oh!

PawelPlesniak commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

MRiganSUSX commented Aug 5, 2026

Description

Type of change

Change log

Suggested manual testing checklist

Developer checklist

Prior to marking this as "Ready for Review"

Final checklist prior to marking this as "Ready for Review"

Reviewer checklist

Prior to merging

Notification message for a Slack channel

For an single merge that changes the user workflow

For co-ordinated merge

Uh oh!

PawelPlesniak Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

PawelPlesniak Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

PawelPlesniak Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

PawelPlesniak commented Aug 6, 2026

Uh oh!

PawelPlesniak commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants