Added an integtest to demonstrate user-specified control applications... by bieryAtFnal · Pull Request #997 · DUNE-DAQ/drunc

bieryAtFnal · 2026-08-05T17:47:04Z

... and updated the existing integtests to react to changes in the integrationtest infrastructure.

Description

These changes are part of the work to add support for multiple user-specified application in our integration tests. The umbrella PR for these changes is DUNE-DAQ/integrationtest#168, and that PR has suggested instructions for testing these changes. All tests of the changes in this repository will need to include the changes in the integrationtest repo.

One of the reasons for creating this PR in "draft" mode is that there are still some changes needed in the process_manager_test.py in order to get it to work.

Type of change

New feature / enhancement

List of required branches from other repositories

integrationtest kbiery/multi_ctrl_proc_support

Change log

The two existing integtests have been updated, and a new integtest has been added.

Suggested manual testing checklist

Please see DUNE-DAQ/integrationtest#168.

Prior to marking this as "Ready for Review"

Tests ran on: WHAT HOSTNAME from release RELEASE_NAME

Unit tests - some tests can't be ran on the CI. This is documented. If this PR checks a feature that can't be tested with CI, this has been marked appropriately.

Integration tests - the daqsystemtest_integtest_bundle requires a lot of resources, and connections to the EHN1 infrastructure. Check the cross referenced list if you can't run these. The developer needs to run at least the .

Unit tests (pytest --marker) passed
- With relevant marker
- Without marker
Integration tests passed
- Only daqsystemtest_integtest_bundle.sh -k minimal_system_quick_test.py
- Full daqsystemtest_integtest_bundle.sh
Testing skipped as there are no core code changes in this PR, this only relates to documentation/CI workflows
Drunc integration tests pass (./scripts/drunc_integtest_bundle.sh)

Final checklist prior to marking this as "Ready for Review"

Code is clearly commented.
New unit tests have been added, or is documented in # ISSUE NUMBER
A suitable reviewer has been chosen from this list.

Reviewer checklist

This branch has been rebased with develop prior to testing.
Suggested manual tests show changes.
CI workflows fails documented (if present)
Integration tests passed (on either np0x or IC HEP clusters)
- Use the following guidelines to determine which of the integration tests you need to run
  - You do not need to run any integration tests if
    - Code changes are not associated with src/
    - PR changes only affect docstrings
    - In this case, be sure to validate any suggested manual testing.
  - Run only the minimum integration test as daqsystemtest_integtest_bundle.sh -k minimal_system_quick_test.py if
    - PR changes only affect a few log entries
    - PR changes are small, and do not have a large impact on the workflow (use carefully)
  - Otherwise run the full integration test bundle as daqsystemtest_integtest_bundle.sh
- What to do if the integration tests fail?
  - Only concern yourself if failures related to drunc are in the log files
  - If non-drunc failure appears:
    - Validate failure in fresh working area
    - Contact Pawel if unsure
If you have ran the full integration test bundle, leave a comment on the PR stating
- Which host the integration tests have ran on
- [Optional] A copy of the test summary
Drunc integration tests pass (scripts/drunc_integtest_bundle.sh)

Once the above boxes are checked, the PR(s) can be merged following the steps below.

Prior to merging

Choose one of the following an complete all substeps

Changes only affect the Run Control, are in a single repository, and do not affect the end user.
- Changes are documented in docstrings and code comments
- Wiki has been updated if architectural or endpoint changes
Otherwise
- Workflow changes demonstrated in the Change Log (if necessary)
- Wiki has been updated (if necessary)
- #dunedaq-integration Slack channel notified (see below)

Once completed, the reviewer can merge the PR.

Notification message for a Slack channel

Note - this should be to #dunedaq-integration for general workflow that isn't during a release candidate period, and to #daq-release-prep otherwise.

For an single merge that changes the user workflow

The CCM WG has an isolated PR ready to merge that affects user workflows. The PR is:

_URL_

I will leave time for any comments, otherwise will merge these at the end of the work day _Insert your time zone_.

For co-ordinated merge

The CCM WG has a set of co-ordinated merges ready to merge. The PRs are:

_URL_

_URL_


I will leave time for any comments, otherwise will merge these at the end of the day.

…est repo to support the running of multiple control apps.

…proc_support

…command so that the time in which triggers are enabled is consistently 20 seconds.

…proc_support

…test.py

emmuhamm · 2026-08-06T09:49:18Z

Suggested change

timestamp_pattern = re.compile(r"\[(\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) UTC\]")

The ^ anchors the regex to the beginning of the string.

Since with the new changes, the captured line is now drunc-unified-shell > [2026/08/06 09:31:55 UTC] INFO commands.py:52 drunc.controller.iface Command wait running for 10 seconds., so the regex can't find the date/time since its no longer at the beginning.

Removing the ^ relaxes this constraint, and the tests now pass :)

emmuhamm · 2026-08-06T10:48:12Z

    print(banner_line)

-    assert run_dunerc.completed_process.returncode == 0
+    assert run_dunerc.completed_processes["drunc"].returncode == 0


Also a quick comment on this before I forget

#868 (recently merged) has introduced a bunch of new tests to drunc.

When this PR gets updated with respect to develop, can you also do the necessary find/replace on the new tests as well?

…proc_support

bieryAtFnal · 2026-08-06T14:11:19Z

@emmuhamm , thanks for taking a look.

Regarding the command prompt that is now present at the beginning of the output lines and is tripping up the process_manager_test...

thanks for identifying that
as you probably guessed, it is a side effect of the different way that commands are send to the run control apps when the apps are user-specified. The commands are fed into the apps one-at-a-time, compared with being given to the drunc-unified-shell on the command line (batch mode, I think) when users don't specify which apps to run.
I can look into suppressing the command prompt in the console output that is returned to integtest "tests". Should I do that?
are there other new characteristics of the console output that is passed back to the integtest "tests" that you've noticed and would like me to change?

Regarding the new integtests that were added to the drunc repo recently, yes, I will update them to use the new completed_processes list.

emmuhamm · 2026-08-06T14:32:29Z

Hi Kurt, indeed I had a feeling that this was a side effect on how they're sent to the run control apps.

I can look into suppressing the command prompt in the console output that is returned to integtest "tests". Should I do that?

I assume this refers to run_dunerc.completed_processes["drunc"].stdout? If so, my gut says that might be a good idea. Since we call it via completed_process[name] anyway, maybe having the drunc-unified-shell > be removed might be nice. However, if it's harder than expected I wouldn't be too worried about it.

Saying that, I think that the regex in the process manager test should be fixed, which can be done by removing the ^ at the beginning. Can you include this fix in this PR if you don't mind?

are there other new characteristics of the console output that is passed back to the integtest "tests" that you've noticed and would like me to change?

Not that I've noticed. All the tests I've ran seem to run fine, and the PM test that failed I would attribute to a too strict regex anyway. As they are now, they look good to me.

…ing drunc output timestamps at exactly the start of a line.

…leted_processes' status list now that we support multiple possible control apps.

bieryAtFnal · 2026-08-06T18:11:00Z

I've made both changes that we talked about regarding the initial failure of the process_manager_test...

I modified the integrationtest infrastructure to strip off the command prompt in the console output
I removed the caret at the front of the regex in process_manager_test

Independent of that, I have updated the new failure_mode_* integtests in the drunc repo so that they use the completed_processes list of status information.

I'm still running tests, so this is just a status update...

…proc_support

Kurt Biery added 9 commits July 27, 2026 15:58

Initial changes to integtests to react to changes in the integrationt…

f3a19ef

…est repo to support the running of multiple control apps.

minor cleanup in basic_multi_app_test.py

d5ce74a

Merge remote-tracking branch 'origin/develop' into kbiery/multi_ctrl_…

b658b64

…proc_support

Restored the pmshell:ps command in the basic_multi_app_test.py test

6803e65

moved the drunc:disable-triggers command earlier than the pmshell:ps …

7f21c2a

…command so that the time in which triggers are enabled is consistently 20 seconds.

Merge remote-tracking branch 'origin/develop' into kbiery/multi_ctrl_…

102db1c

…proc_support

added some comments for the DAQSessionIngredients in basic_multi_app_…

e0b6b31

…test.py

renamed basic_multi_app_test.py as basic_multiapp_test.py

01d03de

Removed debug statement from integ_test_utils.py

9ab3ae9

bieryAtFnal mentioned this pull request Aug 5, 2026

Added support for multiple user-specified applications to be run in integration tests DUNE-DAQ/integrationtest#168

Draft

5 tasks

bieryAtFnal requested review from PawelPlesniak and emmuhamm August 5, 2026 17:53

Added some introduction text to the top of basic_multiapp_test.py

2904e75

emmuhamm added the Testing label Aug 6, 2026

This was linked to issues Aug 6, 2026

Write tests for multi-user process manager usage #908

Open

Write tests for the split shell functionality #907

Open

emmuhamm reviewed Aug 6, 2026

View reviewed changes

Merge remote-tracking branch 'origin/develop' into kbiery/multi_ctrl_…

1d550b9

…proc_support

This was unlinked from issues Aug 6, 2026

Write tests for the split shell functionality #907

Open

Write tests for multi-user process manager usage #908

Open

Kurt Biery added 2 commits August 6, 2026 12:54

Modified process_manager_test.py so that it is less strict about find…

46c307b

…ing drunc output timestamps at exactly the start of a line.

Updated the failure_mode_* integration tests to make use of the 'comp…

4b1683e

…leted_processes' status list now that we support multiple possible control apps.

PawelPlesniak mentioned this pull request Aug 9, 2026

Add client-side rich error interceptor #995

Open

25 tasks

emmuhamm mentioned this pull request Aug 9, 2026

Add split shell PMaaS integtests #1001

Open

20 tasks

Merge remote-tracking branch 'origin/develop' into kbiery/multi_ctrl_…

f5a7258

…proc_support

Navigation Menu

Uh oh!

Added an integtest to demonstrate user-specified control applications... - #997

Added an integtest to demonstrate user-specified control applications...#997
bieryAtFnal wants to merge 14 commits into
developfrom
kbiery/multi_ctrl_proc_support

bieryAtFnal commented Aug 5, 2026 •

edited

Loading

Uh oh!

emmuhamm Aug 6, 2026

Uh oh!

emmuhamm Aug 6, 2026

Uh oh!

bieryAtFnal commented Aug 6, 2026

Uh oh!

emmuhamm commented Aug 6, 2026

Uh oh!

bieryAtFnal commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


	timestamp_pattern = re.compile(r"\[(\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) UTC\]")

Uh oh!

Conversation

bieryAtFnal commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

List of required branches from other repositories

Change log

Suggested manual testing checklist

Prior to marking this as "Ready for Review"

Final checklist prior to marking this as "Ready for Review"

Reviewer checklist

Prior to merging

Notification message for a Slack channel

For an single merge that changes the user workflow

For co-ordinated merge

Uh oh!

emmuhamm Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

emmuhamm Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

bieryAtFnal commented Aug 6, 2026

Uh oh!

emmuhamm commented Aug 6, 2026

Uh oh!

bieryAtFnal commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bieryAtFnal commented Aug 5, 2026 •

edited

Loading