Handling exceptions when applications do not report errors but time out in stateful transitions by PawelPlesniak · Pull Request #868 · DUNE-DAQ/drunc

PawelPlesniak · 2026-03-31T12:45:31Z

Description

Fixes #803
Fixes #869
Fixes #911
Fixes #687

If a segment does not reach the target state, it is marked as in error, and the timeout is logged in the relevant server.
Also defines a set of configurations constructed to fail, and defines a set of unit tests to demonstrate this behaviour.
Error recovery with the supervisor will address what happens if an application completes this outside of the designated window. This is defined in #840

Type of change

List of required branches from other repositories

Requires DUNE-DAQ/druncschema#87

Change log

Addressed issues

Now when an application does not complete its transition in the allocated timeout, the session is put into an error state.
Defines a new click command log (implemented in code as def log_on_server to prevent naming conflicts with logging.Loggers defined as log) that allows the user to send a message to the relevant controller or process manager to put into their log file with a selectable severity level.

Other new features

When running in batch mode, the user can start the run with --no-stop-error-batch-mode. Should an error state be encountered, the session is not terminated, and commands can continue to be executed. This is primarily intended for use with integration tests, such that we can inspect the ps and status tables.

Integration tests for failure modes

Defines a set of intentionally failing configurations in config/tests/failure-testing.data.xml, which contain configurations with a set of fake-daq-apps that fail at configurable points in the session lifecycle. These configurations are

ft-death-on-boot-nest-app - this kills a nested application (2 segments deep) on boot.
ft-death-on-boot-top-app - this kills the top application on boot.
ft-death-post-boot-nest-app - this kills a nested application (2 segments deep) after boot, before applications are marked as ready.
ft-death-post-boot-top-app - this kills the top application after boot, before applications are marked as ready.
ft-fsm-cmd-timeout-nest-app - this times out an FSM transition on a nested application.
ft-fsm-cmd-timeout-top-app - this times out an FSM transition on the top application.
ft-fsm-cmd-death-nest-app - this kills a nested fake daq app during a FSM transition.
ft-fsm-cmd-death-top-app - this kills the top fake daq app during a FSM transition.

The differences between the on-boot and post-boot are useful to see, as the post-boot configs allow the applications that die to register themselves on the connectivity service before dying, hence there is different behaviour.

The differences between the nest-app and top-app allow the user to visualize the effect of a nested hierarchy on the control tree, and how the releveant children applications register themselves to the segment parents. The user shoudl expect to see the majority of the apps in the status table on boot when using a nest config, and only the top two apps when using the top config.

Note that these configurations use the FSMconfiguration_noAction state machine, and will not behave fulliy in the standard way. An example includes not being able to reach the running state as there is no run number allocated, which will be reported as a generic gRPC error.

These tests have been integrated into the unit test framework, which can be executed as (from the local drunc root)

source scripts/setup_drunc_config_path.sh
pytest src/drunc/integtest/<integ_test_file_name>

It is recommended for the tester to use --integtest-verbosity 5 to see what is happening during the process, and if more details are wanted from the pytest side, the user can run the tests as pytest -s to show what would be printed.

Suggested manual testing checklist

Run each of the integration tests as defined above.

Developer checklist

Prior to marking this as "Ready for Review"

Tests ran on: np04-srv-029 from release NFD_DEV_260709_A9.

Unit tests - some tests can't be ran on the CI. This is documented. If this PR checks a feature that can't be tested with CI, this has been marked appropriately.

Integration tests - the daqsystemtest_integtest_bundle requires a lot of resources, and connections to the EHN1 infrastructure. Check the cross referenced list if you can't run these. The developer needs to run at least the .

Unit tests (pytest --marker) passed
- With relevant marker
- Without marker
Integration tests passed
- Only daqsystemtest_integtest_bundle.sh -k minimal_system_quick_test.py
- Full daqsystemtest_integtest_bundle.sh
Testing skipped as there are no core code changes in this PR, this only relates to documentation/CI workflows

Final checklist prior to marking this as "Ready for Review"

Code is clearly commented.
New unit tests have been added, or is documented in # ISSUE NUMBER
A suitable reviewer has been chosen from this list.

Reviewer checklist

This branch has been rebased with develop prior to testing.
Suggested manual tests show changes.
CI workflows fails documented (if present)
Integration tests passed
- Only concern yourself if failures related to drunc are in the log files
- If non-drunc failure appears:
  - Validate failure in fresh working area
  - Contact Pawel if unsure

Once the features are validated and both the unit and integration tests pass, the PRs is ready to be merged.

Prior to merging

Choose one of the following an complete all substeps

Changes only affect the Run Control, are in a single repository, and do not affect the end user.
- Changes are documented in docstrings and code comments
- Wiki has been updated if architectural or endpoint changes
Otherwise
- Workflow changes demonstrated in the Change Log (if necessary)
- Wiki has been updated (if necessary)
- #daq-sw-librarians Slack channel notified (see below)

Once completed, the reviewer can merge the PR.

Notification message for a Slack channel

Note - this should be to #dunedaq-integration for general workflow that isn't during a release candidate period, and to #daq-release-prep otherwise.

For an single merge that changes the user workflow

The CCM WG has an isolated PR ready to merge that affects user workflows. The PR is:

_URL_

I will leave time for any comments, otherwise will merge these at the end of the work day _Insert your time zone_.

For co-ordinated merge

The CCM WG has a set of co-ordinated merges ready to merge. The PRs are:

_URL_

_URL_


I will leave time for any comments, otherwise will merge these at the end of the day.

PawelPlesniak · 2026-03-31T13:30:45Z

In the case where a second application also fails to complete a transition in time, the same error gets thrown. This is likely caused by the nested structure, and the fact that there are multiple layers to this configuration. A robust solution to this problem will take longer to achieve, but I will continue working on it.

…to agreement with SWIT as to where to host these configs)

…M timeouts and death on FSM, REQUIRES TESTING

PawelPlesniak · 2026-06-05T13:58:39Z

Log files do not contain wierd characters from redirecting rich ASCII output to a file with colors
Top app config failure not working?

…to be propagated down to the actual failing app level

PawelPlesniak · 2026-06-08T17:01:12Z

Stream handler has no logging time zone, file name, line number, etc?
Duplicate logs of apps failing
Remove old logs used for debugging
Add druncschema dependency to this PR log
Integrate the failure mode testing into the unit tests

…ansition

…ning these tests

Co-authored-by: Emir Muhammad <49058518+emmuhamm@users.noreply.github.com>

…ithub.com:DUNE-DAQ/drunc into PawelPlesniak/IncompleteStatefulCommandTransition

PawelPlesniak · 2026-08-03T16:03:15Z

I will rerun the integration tests tomorrow in both drunc and daqsystemtest, lets get this PR merged and put away

PawelPlesniak · 2026-08-04T08:41:45Z

All new drunc integration tests have passed

pytest -s src/drunc/integtest/failure_mode_* --integtest-verbosity 5
...
============ 49 passed in 653.81s (0:10:53) ===============

All previous drunc integration tests have also passed.

…g another issue with this PR

PawelPlesniak · 2026-08-04T10:10:02Z

Unit tests now pass

234 passed, 8 skipped, 1 xfailed, 108 warnings in 79.11s (0:01:19)

Integration tests passed on np04-srv-028 with NFD_DEV_260803_A9

+++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++ SUMMARY ++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++

Tue Aug  4 12:09:41 PM CEST 2026
Log file is: /tmp/pytest-of-pplesnia/dunedaq_integtest_bundle_20260804112614.log

⮕ Running daqsystemtest/3ru_1df_multirun_test.py ⬅
======================== 6 passed ✅ in 251.70s (0:04:11) =========================
⮕ Running daqsystemtest/3ru_3df_multirun_test.py ⬅
======================== 6 passed ✅ in 249.92s (0:04:09) =========================
⮕ Running daqsystemtest/disabled_tpg_test.py ⬅
========================= 3 passed ✅ in 61.04s (0:01:01) =========================
⮕ Running daqsystemtest/example_system_test.py ⬅
======================== 12 passed ✅ in 244.42s (0:04:04) ========================
⮕ Running daqsystemtest/fake_data_producer_test.py ⬅
======================== 6 passed ✅ in 237.60s (0:03:57) =========================
⮕ Running daqsystemtest/long_window_readout_test.py ⬅
============================== 1 skipped 🟡 in 1.39s ==============================
⮕ Running daqsystemtest/minimal_system_quick_test.py ⬅
============================== 4 passed ✅ in 54.03s ==============================
⮕ Running daqsystemtest/readout_type_scan_test.py ⬅
======================== 33 passed ✅ in 609.40s (0:10:09) ========================
⮕ Running daqsystemtest/sample_ehn1_multihost_test.py ⬅
======================== 4 skipped 🟡 in 126.48s (0:02:06) ========================
⮕ Running daqsystemtest/small_footprint_quick_test.py ⬅
============================== 3 passed ✅ in 55.55s ==============================
⮕ Running daqsystemtest/tpg_state_collection_test.py ⬅
======================== 5 passed ✅ in 108.56s (0:01:48) =========================
⮕ Running daqsystemtest/tpreplay_test.py ⬅
======================== 6 passed ✅ in 167.04s (0:02:47) =========================
⮕ Running daqsystemtest/tpstream_writing_test.py ⬅
======================== 4 passed ✅ in 108.24s (0:01:48) =========================
⮕ Running daqsystemtest/trigger_bitwords_test.py ⬅
======================== 18 passed ✅ in 312.92s (0:05:12) ========================

PawelPlesniak · 2026-08-04T10:12:52Z

@emmuhamm please review the unresolved comment replies, the rest has been addressed

jamesturner246 · 2026-08-04T11:34:34Z

+        # Children nodes (ignore exclusion).
+        child_list = self.address_target_path(
+            request.target,
+            request.execute_on_all_subsequent_children_in_path,
+            include_excluded_nodes=False,
+        )


Just saw this flash past my screen. NB with include_excluded_nodes=False, you're actually not including excluded nodes, contrary to the comment.

ED: I realise now my (ignore exclusion) comment is kind of ambiguous. Just checking intent.

Thanks James! Either way, we want to include excluded nodes, as the log_on_server command is not stateful

…ansition

emmuhamm

Thanks for addressing my comments, these look great to me! Theres a thread that I've pinged you on since you explicitly asked to leave that unresolved, and another minor comment about updating the code comments, but none of them should block this going in.

Theres also this request you mentioned:

Note to self - requested from MRiganSUSX , list the application that did not complete in time, and tell the user to look at the logs of that specific one

Has this been done? (I haven't dug in the code yet, and might be easier to ask you to confirm >.<)

Aside from that, the only thing I'll ask for is:

a minor conflict that needs to be resolved
pytest (done!)
drunc integtest (done!)
full daqsystemtest (done!)

so after the conflict I'd be happy with evidence that both the pytest, local drunc integtest, and msqt passes and we can get this in!

emmuhamm · 2026-08-04T12:12:38Z

 _PS_COLUMNS = ["session", "friendly_name", "user", "host", "uuid", "alive", "exit_code"]
 _STATUS_COLUMNS = [
    "name",
    "info",
    "state",
    "substate",
    "in_error",
    "included",
    "endpoint",
 ]
 _EXEC_REPORT_COLUMNS = ["name", "command_execution", "fsm_transition"]


Yeah thats fair. The comment was mainly focused on how I'm not a fan of my own design choices, primarily having to hardcode status columns in actual tests.

Eg. we shouldn't ever do table['state'] when writing the tests themselves, since this is prone to error (eg if the name 'state' changes in the future or whatever).

Personally it would be better if when we grab the table we can do a table.state when writing the tests, and abstract away the implementation to the testing utils.

Now obviously this is something far beyond the scope of this PR, but this is something to think about.

emmuhamm · 2026-08-04T12:17:11Z

+pytestmark = pytest.mark.skipif(
+    drunc_missing,
+    reason="drunc is not present in DUNEDAQ_DB_PATH, skipping drunc integration tests",
+)


@PawelPlesniak pinging you on this again since you explicitly asked to leave this unresolved. Is this thread fine now?

emmuhamm · 2026-08-04T12:26:51Z

+    # successful and we can return, otherwise, we will log the error and place the
+    # session in an error state if required.
+    in_error_state = obj.get_driver("controller").status().status.in_error
+    if not in_error_state and not put_in_error_state:


Thanks for adding the comments! A slight request on the comments:

# Determine whether the session should be placed into an error state. See usage for examples on why such a session _should_ be placed into the error state.

Its a lot of handholding but I think this is important, at least it'll tell a developer what the variables does and tells them to read the code for a few examples of why put_in_error_state needs to be used.

PawelPlesniak · 2026-08-04T12:34:59Z

Theres a thread that I've pinged you

Thanks

Has this been done? (I haven't dug in the code yet, and might be easier to ask you to confirm >.<)

Yep

a minor conflict that needs to be resolved

Also done ;)

Other comments will be addressed in the relevant threads now

Thanks!

PawelPlesniak · 2026-08-04T14:13:20Z

At this stage, I think we're coming to a close, the last topic is the discussion on the in_error_state flag

emmuhamm

At this stage, I think we're coming to a close, the last topic is the discussion on the in_error_state flag

Agree! Thats why I'm going to approve this right now so you can merge it whenever this is resolved but so it isn't blocked on my availability.

In any case I've seen all the code already and whatever's gonna change is just a comment. So as long as the aformentioned checks are done this can go in.

Thanks for this nice work my guy!

Generating an environment for which the issue can be recreated

a7ba095

PawelPlesniak changed the title ~~Generating an environment for which the issue can be recreated~~ Handling exceptions when applications do not report errors but time out in stateful transitions Mar 31, 2026

WIP

c50fb9f

PawelPlesniak changed the base branch from prep-release/fddaq-v5.6.0 to develop June 4, 2026 16:05

PawelPlesniak added 15 commits June 4, 2026 18:08

Merging

db1574e

Resovling duplicate config

456a461

Adding drunc to the DUNEDAQ_DB_PATH (should get replaced but subject …

785e98b

…to agreement with SWIT as to where to host these configs)

Splitting out the failure mode testing into a separate file

7da158d

Making the configs actually pass oks_dump with the DB path

f6f8c62

Remmoving duplicate confs

8859c4b

Constructed tests for death on boot and post boot, as well as long FS…

234f185

…M timeouts and death on FSM, REQUIRES TESTING

Adding death cases

1bf07c6

Clearer conf param names

34d45b9

Configs developed, time to test

c4ead7f

Remove dupe var definitions

cf4ddb8

Fixing the broken session -> OKS var links

2a9348a

Included the configs, damned be gitignore

c8fa56e

Vars rechecked

1323cce

Moving point of timeout

abde856

PawelPlesniak added 6 commits June 5, 2026 16:59

WIP

468bef9

Root controller is now marked as in error on timeout, but this needs …

92a5314

…to be propagated down to the actual failing app level

Printing the status after marking an error state

1aa6d05

Adding log RPC

2917d12

Logs on remote server functional

899c510

WIP

6554774

PawelPlesniak added 2 commits June 9, 2026 14:36

WIP continuned

b169b22

Logfiles now have a proper format

57a8cfa

PawelPlesniak and others added 7 commits August 3, 2026 12:01

Merge branch 'develop' into PawelPlesniak/IncompleteStatefulCommandTr…

f23c153

…ansition

Centralizing the DB path check logic

5e91435

Adding the relevant docstrings

125bd59

Renaming the tests for naming consistency

f1df46d

Documentation and comment addressing

4ebe2d0

Making it clearer which application the user should focus on when run…

f3195c3

…ning these tests

Update src/drunc/integtest/failure_mode_death_on_boot_nest_app_test.py

e7ee9a7

Co-authored-by: Emir Muhammad <49058518+emmuhamm@users.noreply.github.com>

PawelPlesniak mentioned this pull request Aug 3, 2026

[Feature]: Simplify the failure mode tests #993

Open

PawelPlesniak added 4 commits August 3, 2026 17:47

Simplifying the pytest

2f14250

Changing dependency from external to internal

ae81235

Merge branch 'PawelPlesniak/IncompleteStatefulCommandTransition' of g…

19cadc1

…ithub.com:DUNE-DAQ/drunc into PawelPlesniak/IncompleteStatefulCommandTransition

Simplifying logic

745f5af

Minor updates to logging and testing output

e6ad1ab

Correcting pytests, failures associated with bad merges, and resolvin…

45cba8d

…g another issue with this PR

jamesturner246 reviewed Aug 4, 2026

View reviewed changes

PawelPlesniak added 2 commits August 4, 2026 14:16

Changing excluded nodes to be included

88fb003

Merge branch 'develop' into PawelPlesniak/IncompleteStatefulCommandTr…

c4fa1f2

…ansition

emmuhamm reviewed Aug 4, 2026

View reviewed changes

PawelPlesniak mentioned this pull request Aug 4, 2026

[Feature]: Type checking with mypy and docstrings #828

Open

PawelPlesniak added 3 commits August 4, 2026 15:06

Remvoing resisudal time.sleep

5fc0a4d

Genearlizing the drunc DUNEDAQ_DB_PATH check, and fixing the ruff config

646bade

Adding a ternary preference

4535848

PawelPlesniak mentioned this pull request Aug 4, 2026

The testing framework #958

Open

emmuhamm approved these changes Aug 4, 2026

View reviewed changes

Uh oh!

Conversation

PawelPlesniak commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

List of required branches from other repositories

Change log

Addressed issues

Other new features

Integration tests for failure modes

Suggested manual testing checklist

Developer checklist

Prior to marking this as "Ready for Review"

Final checklist prior to marking this as "Ready for Review"

Reviewer checklist

Prior to merging

Notification message for a Slack channel

For an single merge that changes the user workflow

For co-ordinated merge

Uh oh!

PawelPlesniak commented Mar 31, 2026

Uh oh!

PawelPlesniak commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

PawelPlesniak commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

PawelPlesniak commented Aug 3, 2026

Uh oh!

PawelPlesniak commented Aug 4, 2026

Uh oh!

PawelPlesniak commented Aug 4, 2026

Uh oh!

PawelPlesniak commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jamesturner246 Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PawelPlesniak Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

emmuhamm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

emmuhamm Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

emmuhamm Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

emmuhamm Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

PawelPlesniak commented Aug 4, 2026

Uh oh!

PawelPlesniak commented Aug 4, 2026

Uh oh!

emmuhamm left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

PawelPlesniak commented Mar 31, 2026 •

edited

Loading

PawelPlesniak commented Jun 5, 2026 •

edited

Loading

PawelPlesniak commented Jun 8, 2026 •

edited

Loading

PawelPlesniak commented Aug 4, 2026 •

edited

Loading

jamesturner246 Aug 4, 2026 •

edited

Loading