Handling exceptions when applications do not report errors but time out in stateful transitions by PawelPlesniak · Pull Request #868 · DUNE-DAQ/drunc

PawelPlesniak · 2026-03-31T12:45:31Z

Description

Fixes #803
Fixes #911

If a segment does not reach the target state, it is marked as in error, and the timeout is logged in the relevant server.
Also defines a set of configurations constructed to fail, and defines a set of unit tests to demonstrate this behaviour.
Error recovery with the supervisor will address what happens if an application completes this outside of the designated window. This is defined in #840

Type of change

List of required branches from other repositories

Requires DUNE-DAQ/druncschema#87

Change log

Addressed issues

Now when an application does not complete its transition in the allocated timeout, the session is put into an error state.
Defines a new click command log that allows the user to send a message to the relevant controller or process manager to put into their log file with a selectable severity level.

Other new features

When running in batch mode, the user can start the run with --no-stop-error-batch-mode. Should an error state be encountered, the session is not terminated, and commands can continue to be executed. This is primarily intended for use with integration tests, such that we can inspect the ps and status tables.

Integration tests for failure modes

Defines a set of intentionally failing configurations in config/tests/failure-testing.data.xml, which contain configurations with a set of fake-daq-apps that fail at configurable points in the session lifecycle. These configurations are

ft-death-on-boot-nest-app - this kills a nested application (2 segments deep) on boot.
ft-death-on-boot-top-app - this kills the top application on boot.
ft-death-post-boot-nest-app - this kills a nested application (2 segments deep) after boot, before applications are marked as ready.
ft-death-post-boot-top-app - this kills the top application after boot, before applications are marked as ready.
ft-fsm-cmd-timeout-nest-app - this times out an FSM transition on a nested application.
ft-fsm-cmd-timeout-top-app - this times out an FSM transition on the top application.
ft-fsm-cmd-death-nest-app - this kills a nested fake daq app during a FSM transition.
ft-fsm-cmd-death-top-app - this kills the top fake daq app during a FSM transition.

The differences between the on-boot and post-boot are useful to see, as the post-boot configs allow the applications that die to register themselves on the connectivity service before dying, hence there is different behaviour.

The differences between the nest-app and top-app allow the user to visualize the effect of a nested hierarchy on the control tree, and how the releveant children applications register themselves to the segment parents. The user shoudl expect to see the majority of the apps in the status table on boot when using a nest config, and only the top two apps when using the top config.

Note that these configurations use the FSMconfiguration_noAction state machine, and will not behave fulliy in the standard way. An example includes not being able to reach the running state as there is no run number allocated, which will be reported as a generic gRPC error.

These tests have been integrated into the unit test framework, which can be executed as (from the local drunc root)

source scripts/setup_drunc_config_path.sh
pytest src/drunc/integtest/<integ_test_file_name>

It is recommended for the tester to use --integtest-verbosity 5 to see what is happening during the process, and if more details are wanted from the pytest side, the user can run the tests as pytest -s to show what would be printed.

Suggested manual testing checklist

Run each of the integration tests as defined above.

Developer checklist

Prior to marking this as "Ready for Review"

Tests ran on: np04-srv-029 from release NFD_DEV_260709_A9.

Unit tests - some tests can't be ran on the CI. This is documented. If this PR checks a feature that can't be tested with CI, this has been marked appropriately.

Integration tests - the daqsystemtest_integtest_bundle requires a lot of resources, and connections to the EHN1 infrastructure. Check the cross referenced list if you can't run these. The developer needs to run at least the .

Unit tests (pytest --marker) passed
- With relevant marker
- Without marker
Integration tests passed
- Only daqsystemtest_integtest_bundle.sh -k minimal_system_quick_test.py
- Full daqsystemtest_integtest_bundle.sh
Testing skipped as there are no core code changes in this PR, this only relates to documentation/CI workflows

Final checklist prior to marking this as "Ready for Review"

Code is clearly commented.
New unit tests have been added, or is documented in # ISSUE NUMBER
A suitable reviewer has been chosen from this list.

Reviewer checklist

This branch has been rebased with develop prior to testing.
Suggested manual tests show changes.
CI workflows fails documented (if present)
Integration tests passed
- Only concern yourself if failures related to drunc are in the log files
- If non-drunc failure appears:
  - Validate failure in fresh working area
  - Contact Pawel if unsure

Once the features are validated and both the unit and integration tests pass, the PRs is ready to be merged.

Prior to merging

Choose one of the following an complete all substeps

Changes only affect the Run Control, are in a single repository, and do not affect the end user.
- Changes are documented in docstrings and code comments
- Wiki has been updated if architectural or endpoint changes
Otherwise
- Workflow changes demonstrated in the Change Log (if necessary)
- Wiki has been updated (if necessary)
- #daq-sw-librarians Slack channel notified (see below)

Once completed, the reviewer can merge the PR.

Notification message for a Slack channel

Note - this should be to #dunedaq-integration for general workflow that isn't during a release candidate period, and to #daq-release-prep otherwise.

For an single merge that changes the user workflow

The CCM WG has an isolated PR ready to merge that affects user workflows. The PR is:

_URL_

I will leave time for any comments, otherwise will merge these at the end of the work day _Insert your time zone_.

For co-ordinated merge

The CCM WG has a set of co-ordinated merges ready to merge. The PRs are:

_URL_

_URL_


I will leave time for any comments, otherwise will merge these at the end of the day.

PawelPlesniak · 2026-03-31T13:30:45Z

In the case where a second application also fails to complete a transition in time, the same error gets thrown. This is likely caused by the nested structure, and the fact that there are multiple layers to this configuration. A robust solution to this problem will take longer to achieve, but I will continue working on it.

…to agreement with SWIT as to where to host these configs)

…M timeouts and death on FSM, REQUIRES TESTING

PawelPlesniak · 2026-06-05T13:58:39Z

Log files do not contain wierd characters from redirecting rich ASCII output to a file with colors
Top app config failure not working?

…to be propagated down to the actual failing app level

PawelPlesniak · 2026-06-08T17:01:12Z

Stream handler has no logging time zone, file name, line number, etc?
Duplicate logs of apps failing
Remove old logs used for debugging
Add druncschema dependency to this PR log
Integrate the failure mode testing into the unit tests

PawelPlesniak · 2026-07-16T13:16:21Z

Integration tests passed on np04-srv-029

======================== 18 passed in 293.70s (0:04:53) ========================

+++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++ SUMMARY ++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++

Thu Jul 16 03:15:52 PM CEST 2026
Log file is: /tmp/pytest-of-pplesnia/dunedaq_integtest_bundle_20260716143520.log

⮕ Running daqsystemtest/3ru_1df_multirun_test.py ⬅
======================== 6 passed ✅ in 244.30s (0:04:04) =========================
⮕ Running daqsystemtest/3ru_3df_multirun_test.py ⬅
======================== 6 passed ✅ in 243.98s (0:04:03) =========================
⮕ Running daqsystemtest/disabled_tpg_test.py ⬅
============================== 3 passed ✅ in 59.25s ==============================
⮕ Running daqsystemtest/example_system_test.py ⬅
======================== 12 passed ✅ in 234.24s (0:03:54) ========================
⮕ Running daqsystemtest/fake_data_producer_test.py ⬅
======================== 6 passed ✅ in 235.73s (0:03:55) =========================
⮕ Running daqsystemtest/long_window_readout_test.py ⬅
============================== 1 skipped 🟡 in 1.52s ==============================
⮕ Running daqsystemtest/minimal_system_quick_test.py ⬅
============================== 4 passed ✅ in 53.46s ==============================
⮕ Running daqsystemtest/readout_type_scan_test.py ⬅
======================== 33 passed ✅ in 576.37s (0:09:36) ========================
⮕ Running daqsystemtest/sample_ehn1_multihost_test.py ⬅
======================== 4 skipped 🟡 in 84.05s (0:01:24) =========================
⮕ Running daqsystemtest/small_footprint_quick_test.py ⬅
============================== 3 passed ✅ in 51.67s ==============================
⮕ Running daqsystemtest/tpg_state_collection_test.py ⬅
======================== 5 passed ✅ in 107.56s (0:01:47) =========================
⮕ Running daqsystemtest/tpreplay_test.py ⬅
======================== 6 passed ✅ in 118.26s (0:01:58) =========================
⮕ Running daqsystemtest/tpstream_writing_test.py ⬅
======================== 4 passed ✅ in 110.07s (0:01:50) =========================
⮕ Running daqsystemtest/trigger_bitwords_test.py ⬅
======================== 18 passed ✅ in 293.70s (0:04:53) ========================

…ansition

PawelPlesniak · 2026-07-17T14:42:06Z

Duplicate code is present in develop between send_msg and log_on_server, resolving this to log_on_server, see #888 for more details

PawelPlesniak · 2026-07-17T14:58:40Z

@emmuhamm should be good to go

…ansition

emmuhamm · 2026-07-23T10:34:34Z

Note that for the review process, this PR and the dependency is up to date with develop as of today's (23 june) nightly.

A few changes that were implemented are porting over some of the new functionality that now exists in develop (particularly about how all lines should have stripped_ansi now) and minor changes to get some of the other tests working. Including increasing the widths of the table. as well as changing the column names.

I'll proceed to give comments, but this may come in bits and pieces

emmuhamm

Review 1/?

Ran out of bandwidth for now but this should be most of my comments for the testing side of things.

Pretty minor comments for this PR, some could be defferred for a bigger issue. Importantly, all the tests run just fine

emmuhamm · 2026-07-21T10:34:03Z

+pytestmark = pytest.mark.skipif(
+    drunc_missing,
+    reason="drunc is not present in DUNEDAQ_DB_PATH, skipping drunc integration tests",
+)


This error message should also say how to fix this. (this applied to all the files)

This is a good point to raise. However, the way in which the configuration file is packaged for retrieval by the integration tests defined within python-only DD repos is still under discussion. Please leave this unresolved so this can be addressed ASAP.

emmuhamm · 2026-07-21T10:37:06Z

+drunc_missing = not any(
+    "drunc" == segment for path in db_path_env.split(":") for segment in path.split("/")
+)


For posterity: why is this necessary?

This is the way to check if drunc is included as part of the DUNEDAQ_DB_PATH env var, so that the path to the configuration file can actually be matched (and the session can be run). This need to be explicitly stated as equality, as initial implementations matched this for druncschema, which did caused failure of the integration tests (the session wouldn't run as it wouldn't find the config, and would just print a stack of errors)

emmuhamm · 2026-07-23T10:50:09Z

+    except FileNotFoundError:
+        print(f"Error: {file} not found.")
+        return False


Do we want to return false here or throw? Feels like in a testing environment if we cant find the file it should throw

This should be left as a boolean return, mainly for cleaner output. The way that the test is currently structured is on a query base, where we check whether the line exists, and if it does not, then we return. Both can achieve the same functionaltiy, but I think this is cleaner. Should this fail in the future, when running with pytest -s (-s allows seeing the print statements in the tests), the error will be a one line solution, rather than a stacktrace.

emmuhamm · 2026-07-23T10:54:02Z

 _PS_COLUMNS = ["session", "friendly_name", "user", "host", "uuid", "alive", "exit_code"]
 _STATUS_COLUMNS = [
    "name",
    "info",
    "state",
    "substate",
    "in_error",
    "included",
    "endpoint",
 ]
 _EXEC_REPORT_COLUMNS = ["name", "command_execution", "fsm_transition"]


After the big developments of this tests, im starrting to think we should improve this part. Specifically, the columns themselves are called in the tests 'eg check if [substate] exists'

We should definitely strive to improve this, but this is beyond the scope of this PR.

If you agree, can convert this to an issue?

Sorry the question here is unclear to me, what would you propose changing with these variables?

emmuhamm · 2026-07-23T10:56:53Z

+#! Replace this with a generic one
+def get_rows_by_friendly_name_from_ps_table(
    ps_table: list[dict[str, str]], friendly_name: str
 ) -> list[dict[str, str]]:
    """Return all rows whose `friendly_name` matches exactly after stripping."""
    return [row for row in ps_table if row["friendly_name"].strip() == friendly_name]


+def get_rows_by_name_from_status_table(
+    status_table: list[dict[str, str]], name: str
+) -> list[dict[str, str]]:
+    """Return all rows whose `Name` matches exactly after stripping."""
+    return [row for row in status_table if row["name"].strip() == name]
+


The two functions are nearly identical. Suggest

def get_rows_by_name_from_table(table: list[dict[str,str]], name:str, row_name : str) -> list: return [row for row in table if row[row_name].strip() == name]

Would this be easy to implement? If not, happy to fold this into the issue i mentioned above about the columns

The issue is that the ps table lists the process name as friendly_name, and the status table lists the process name as name. I think we could do this, but this should be a separate PR, including changing the actual ps table process name to just name and not friendly_name

PawelPlesniak · 2026-07-23T15:55:26Z

Thanks @emmuhamm I will look tomorrow
Note to self - requested from @MRiganSUSX , list the application that did not complete in time, and tell the user to look at the logs of that specific one

Co-authored-by: Emir Muhammad <49058518+emmuhamm@users.noreply.github.com>

PawelPlesniak

Sorry, for some reason parts of these comments were just comment replies, others were a second review.

PawelPlesniak · 2026-07-24T12:27:05Z

 _PS_COLUMNS = ["session", "friendly_name", "user", "host", "uuid", "alive", "exit_code"]
 _STATUS_COLUMNS = [
    "name",
    "info",
    "state",
    "substate",
    "in_error",
    "included",
    "endpoint",
 ]
 _EXEC_REPORT_COLUMNS = ["name", "command_execution", "fsm_transition"]


Sorry the question here is unclear to me, what would you propose changing with these variables?

PawelPlesniak · 2026-07-24T12:28:43Z

+#! Replace this with a generic one
+def get_rows_by_friendly_name_from_ps_table(
    ps_table: list[dict[str, str]], friendly_name: str
 ) -> list[dict[str, str]]:
    """Return all rows whose `friendly_name` matches exactly after stripping."""
    return [row for row in ps_table if row["friendly_name"].strip() == friendly_name]


+def get_rows_by_name_from_status_table(
+    status_table: list[dict[str, str]], name: str
+) -> list[dict[str, str]]:
+    """Return all rows whose `Name` matches exactly after stripping."""
+    return [row for row in status_table if row["name"].strip() == name]
+


The issue is that the ps table lists the process name as friendly_name, and the status table lists the process name as name. I think we could do this, but this should be a separate PR, including changing the actual ps table process name to just name and not friendly_name

PawelPlesniak · 2026-07-24T12:29:19Z

+    except FileNotFoundError:
+        print(f"Error: {file} not found.")
+        return False


This should be left as a boolean return, mainly for cleaner output. The way that the test is currently structured is on a query base, where we check whether the line exists, and if it does not, then we return. Both can achieve the same functionaltiy, but I think this is cleaner. Should this fail in the future, when running with pytest -s (-s allows seeing the print statements in the tests), the error will be a one line solution, rather than a stacktrace.

…lication times out

…ithub.com:DUNE-DAQ/drunc into PawelPlesniak/IncompleteStatefulCommandTransition

emmuhamm

Hello hello here's another round of reviews. Gone through most of the log code and mostly appears fine. Only got a few comments on that bit

emmuhamm · 2026-07-24T15:40:56Z

+    log = get_logger("unified_shell.log_on_server")
+    log.debug("Logging message to server(s)...")
+
+    if target_server in ["", "process_manager"]:


log --target-server df-controller "hello3"
I would expect this to send a hello3 to the 'df-controller' server. But this doesnt work.

Maybe a smarter system that can distribute it?

(Maybe beyond the scope of this PR but useful to think about)

emmuhamm · 2026-07-24T15:51:20Z

+    def log_on_server(
+        self,
+        text: str,
+        severity: str = "INFO",
+        target: str = "",
+        execute_along_path: bool = False,
+        execute_on_all_subsequent_children_in_path: bool = True,
+    ) -> LogOnServerResponse:


emmuhamm · 2026-07-24T15:51:37Z

            flag=ResponseFlag.EXECUTED_SUCCESSFULLY,
        )
+
+    def log_on_server(


emmuhamm · 2026-07-24T15:52:04Z

    execute_on_all_subsequent_children_in_path: bool,
    extended: bool,
 ) -> None:
+    obj.log.warning(f"Getting status for target '{target}'...")


why is this a warning?

Generating an environment for which the issue can be recreated

a7ba095

PawelPlesniak changed the title ~~Generating an environment for which the issue can be recreated~~ Handling exceptions when applications do not report errors but time out in stateful transitions Mar 31, 2026

WIP

c50fb9f

PawelPlesniak changed the base branch from prep-release/fddaq-v5.6.0 to develop June 4, 2026 16:05

PawelPlesniak added 15 commits June 4, 2026 18:08

Merging

db1574e

Resovling duplicate config

456a461

Adding drunc to the DUNEDAQ_DB_PATH (should get replaced but subject …

785e98b

…to agreement with SWIT as to where to host these configs)

Splitting out the failure mode testing into a separate file

7da158d

Making the configs actually pass oks_dump with the DB path

f6f8c62

Remmoving duplicate confs

8859c4b

Constructed tests for death on boot and post boot, as well as long FS…

234f185

…M timeouts and death on FSM, REQUIRES TESTING

Adding death cases

1bf07c6

Clearer conf param names

34d45b9

Configs developed, time to test

c4ead7f

Remove dupe var definitions

cf4ddb8

Fixing the broken session -> OKS var links

2a9348a

Included the configs, damned be gitignore

c8fa56e

Vars rechecked

1323cce

Moving point of timeout

abde856

PawelPlesniak added 6 commits June 5, 2026 16:59

WIP

468bef9

Root controller is now marked as in error on timeout, but this needs …

92a5314

…to be propagated down to the actual failing app level

Printing the status after marking an error state

1aa6d05

Adding log RPC

2917d12

Logs on remote server functional

899c510

WIP

6554774

PawelPlesniak added 2 commits June 9, 2026 14:36

WIP continuned

b169b22

Logfiles now have a proper format

57a8cfa

PawelPlesniak added 6 commits July 13, 2026 16:51

Pytest fix

0da7956

log_on_server WIP

9c54136

Done?

1b6e2c0

Reducing verboisty of an error message

8543da7

Small fixes

b430dd1

Updating the driver script for integration tests

67f39c4

PawelPlesniak marked this pull request as ready for review July 16, 2026 13:19

PawelPlesniak requested a review from emmuhamm July 16, 2026 13:19

Merge branch 'develop' into PawelPlesniak/IncompleteStatefulCommandTr…

2e2dcab

…ansition

PawelPlesniak added 2 commits July 17, 2026 16:57

Merge complete

bcf5fe7

Ruff linting

c9a9cc4

emmuhamm mentioned this pull request Jul 21, 2026

ps -u in unified shell broken #979

Closed

emmuhamm added 5 commits July 21, 2026 12:34

Merge branch 'develop' into PawelPlesniak/IncompleteStatefulCommandTr…

ddd7860

…ansition

Patch changes from integtest development

2dc0984

Clean up minor issues from other tests

99149a6

Merge branch 'develop' into PawelPlesniak/IncompleteStatefulCommandTr…

ee14f62

…ansition

Fix failures

15d1735

emmuhamm reviewed Jul 23, 2026

View reviewed changes

PawelPlesniak added this to the fddaq-v5.7.0 milestone Jul 23, 2026

Update src/drunc/integtest/test_failure_mode_death_on_boot_nest_app.py

113b82b

Co-authored-by: Emir Muhammad <49058518+emmuhamm@users.noreply.github.com>

PawelPlesniak commented Jul 24, 2026

View reviewed changes

PawelPlesniak added 3 commits July 24, 2026 17:13

Adding requested feature to state what command to execute when an app…

215f0c8

…lication times out

extending tests

c6717ab

Merge branch 'PawelPlesniak/IncompleteStatefulCommandTransition' of g…

291e6cc

…ithub.com:DUNE-DAQ/drunc into PawelPlesniak/IncompleteStatefulCommandTransition

emmuhamm reviewed Jul 24, 2026

View reviewed changes

Uh oh!

Conversation

PawelPlesniak commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

List of required branches from other repositories

Change log

Addressed issues

Other new features

Integration tests for failure modes

Suggested manual testing checklist

Developer checklist

Prior to marking this as "Ready for Review"

Final checklist prior to marking this as "Ready for Review"

Reviewer checklist

Prior to merging

Notification message for a Slack channel

For an single merge that changes the user workflow

For co-ordinated merge

Uh oh!

PawelPlesniak commented Mar 31, 2026

Uh oh!

PawelPlesniak commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

PawelPlesniak commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

PawelPlesniak commented Jul 16, 2026

Uh oh!

PawelPlesniak commented Jul 17, 2026

Uh oh!

PawelPlesniak commented Jul 17, 2026

Uh oh!

emmuhamm commented Jul 23, 2026

Uh oh!

emmuhamm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

PawelPlesniak commented Jul 23, 2026

Uh oh!

PawelPlesniak left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emmuhamm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PawelPlesniak commented Mar 31, 2026 •

edited

Loading

PawelPlesniak commented Jun 5, 2026 •

edited

Loading

PawelPlesniak commented Jun 8, 2026 •

edited

Loading