Introduce new handlers and refactor filters by emmuhamm · Pull Request #37 · DUNE-DAQ/daqpytools

emmuhamm · 2026-01-21T10:52:55Z

Description

Fixes #36

This PR Introduces a couple of changes that fixes #36

Added Throttling Filter (and demonstration)

This dynamically filters out messages if many of them get sent. Logic exactly follows the C++ implementation. Defaults are also the same, with 30 seconds and 30 messages. Note that when adding the Throttle filter using get_daq_logger, it uses a boolean and so the 30s and 30msg can't be changed. This is fine for now but should be easy to make more customisable later.

Added support for Lstdout

In ERS C++, Lstdout is the thread-safe version of stdout. The default logging implementation in Python is poggers as stdout is already threadsafe. Therefore, for this implementation we match HandlerType.Lstdout to just run for the stdout handler.
This is done via stdout_handler.addFilter(HandleIDFilter([HandlerType.Stream, HandlerType.Lstdout])).
Relevant changes in the code so HandleIDFilter supports multiple handlertypes to transmit with

Improves Filtering class logic and inheritance

Refactored the old HandleIDFiltering code to a BaseHandlerFilter class which contains a lot of the checks on when to emit a message or not.
Inherited by HandleIDFilter, makes it much easier to understand
Also Inherited by ThrottleFilter as Throttle is technically a 'HandlerType', to reuse the checking logic

Move away from string based log level matching to enum-based log level matching

When developing, there was a bug where using StreamHandler would break the ERS functionality.
This is because running StreamHandler modifies the record via the LoggingFormatter, and previously the Filter uses level_to_ers_var.get(record.levelname)), which would now contain erroneous whitespace and matching would fail
Moved from "DEBUG" to logging.DEBUG fixes this

Improves logging of the logging framework

in OKS, the ERS variables are of the form "erstrace,throttle,lstdout"
ERSTrace is no longer supported, so a debug message is sent via a logger in daqpytools to reveal this
If the variable is "erstrace,throttle,random", where 'random' is not supported, the logger will send a warning message.

TODO

Fix the comments @emmuhamm left in the code
Run linter

Type of change

Documentation (non-breaking change that adds or improves the documentation)
New feature or enhancement (non-breaking change which adds functionality)
Optimization (non-breaking change that improves code/performance)
Bug fix (non-breaking change which fixes an issue)
Breaking change (whatever its nature)

Testing checklist

Unit tests pass (e.g. dbt-build --unittest)
Minimal system quicktest passes (pytest -s minimal_system_quick_test.py)
Full set of integration tests pass (daqsystemtest_integtest_bundle.sh)
Python tests pass if applicable (e.g. python -m pytest)
Pre-commit hooks run successfully if applicable (e.g. pre-commit run --all-files)

How to test the changes

Check out this branch on the latest nightly
Run:
- daqpytools-logging-demonstrator -r
  - Should see the normal messages
- daqpytools-logging-demonstrator -r --throttle
  - Demonstration of throttling. Should see a bunch of message print out and then throttled, a 30s wait, and then more messages and throttling
- daqpytools-logging-demonstrator -s --handlertypes
  - this enables the stream handler (stdout + stderr)
  - Should see the usual example messages, and then the ERS messages going through
  - You should see anything that has lstdout be printed via the stdout handler
  - NOTE there will be duplicates anytime stdout + stderr is used in the demonstrator (eg in the example messages). This is documented in [Bug]: StreamHandler log levels are weirdly being overridden #44, and beyond the scope of this PR.

Further checks

Code is commented where needed, particularly in hard-to-understand areas
Code style is correct (dbt-build --lint, and/or see https://dune-daq-sw.readthedocs.io/en/latest/packages/styleguide/)
If applicable, new tests have been added or an issue has been opened to tackle that in the future.
(Indicate issue here: # (issue))

src/daqpytools/logging/logger.py

src/daqpytools/logging/handlers.py

emmuhamm · 2026-02-04T11:32:35Z

Hi @PawelPlesniak, in the interest of timing and availability I'll ask for your review today before I leave. The overall functionality exists and works as expected.

Theres a few minor improvements here and there that I've commented about, but they shouldn't affect the functionality. Theres also a couple where I ask for input.

But yeah, wouldbe great if you can take a look at the code already + the functionality, and lmk what you think

PawelPlesniak · 2026-02-13T09:27:27Z

Things to check

Equivalent timestamp format in FormattedRichHandler and ThrottleFilter

Notes

Thanks @emmuhamm this looks like exactly what we want. Few commentts

For this PR, the ThrottleFilter implementation is correct. We will have to discuss with @mroda88 to confirm whether we want customizability (and we'll have to message him through Slack, he ignores Github notifications, as confirmed in a recent meeting). If changes are requested, this will go in a separate PR.
The logging demonstrator should not affect the environment it is run in, this may cause interference with the testing strategy, and we don't want to have to run this demonstrator prior to running anything. To clarify, you have a set of environemnt variables injected (see below). Can you save what they are prior to use, and then export their original values after?
Env vars injected

    os.environ["DUNEDAQ_ERS_WARNING"] = "erstrace,throttle,lstdout"
    os.environ["DUNEDAQ_ERS_INFO"] = "erstrace,throttle,lstdout"
    os.environ["DUNEDAQ_ERS_FATAL"] = "erstrace,lstdout"
    os.environ["DUNEDAQ_ERS_ERROR"] = (
        "erstrace,"
        "throttle,"
        "lstdout,"
        "protobufstream(monkafka.cern.ch:30092)"
    )

Thanks also for the pull request template, that's a good catch.

I will check the Things to check section at the top, and then run some tests, but this logically looks good. In the meantime, can you introduce the stderr handler we have discussed and do the env var recovery, then I will run more tests.

In the meantime, I will review some of your other PRs

emmuhamm · 2026-02-13T09:30:38Z

Hi Pawel, thanks for your review. I will go ahead and do the lstderr stuff.

Also good shout on the environment stuff, I did not consider that when i was writing this. I'll go ahead and make the change, will msg you on slack when this is ready to go again

PawelPlesniak · 2026-02-13T09:57:00Z

The env vars will be allocated to the subsequent PR, testing ongoing

PawelPlesniak · 2026-02-13T10:50:33Z

Follow up comments and question

When using the throttle testing, on boundary conditions (i.e. when reporting the last repeated message), the times are quite similar. It looks like 90 messages were published in 3ms, which is good.

[2026/02/13 10:38:42 UTC] INFO       logging_demonstrator.py:224              daqpytools_logging_demonstrator                    Throttle test 29
[2026/02/13 10:38:42 UTC] INFO       logging_demonstrator.py:224              daqpytools_logging_demonstrator                    Throttle test 40 -- 10 similar messages suppressed, last occurrence was at 2026-02-13 11:38:42.139987
[2026/02/13 10:38:42 UTC] INFO       logging_demonstrator.py:224              daqpytools_logging_demonstrator                    Throttle test 141 -- 100 similar messages suppressed, last occurrence was at 2026-02-13 11:38:42.142920

Also note the filter is nicely implemented, it's good to see that it integrates well with rich.
Can we have the time reported also contain the timezone? This can be copied from the FormattedRichHandler.

Issue #44 would be good to prioritize in preparation for showing in SWIT

emmuhamm · 2026-02-13T10:52:54Z

Can we have the time reported also contain the timezone? This can be copied from the FormattedRichHandler.

Sure, will do so in a bit

Issue #44 would be good to prioritize in preparation for showing in SWIT

#46 ;)

PawelPlesniak · 2026-02-13T12:19:22Z

Can we have the time reported also contain the timezone? This can be copied from the FormattedRichHandler.

Sure, will do so in a bit

Issue #44 would be good to prioritize in preparation for showing in SWIT

#46 ;)

Am reviewing #46 now

emmuhamm self-assigned this Jan 21, 2026

emmuhamm force-pushed the emmuhamm/new-handlers branch from 5a05edb to 4a16756 Compare January 22, 2026 09:28

Base automatically changed from emmuhamm/introduce-ers to develop February 2, 2026 14:53

emmuhamm and others added 9 commits February 2, 2026 16:58

start on new handlers

9f4d04b

Working throttle implementation

df7100f

Throttling works with handlertypes

4153880

throttle supression goes to correct handlers

ee1efa5

Add throttle as filter to getdaqlogger

9cdba81

Add support for lstdout

dc45ca7

Clean up PR

260dbca

Clean up throttle in demonstrator

e1718bf

Fix loglevel bug, cleanup deepcopy

6a60f1e

emmuhamm force-pushed the emmuhamm/new-handlers branch 2 times, most recently from 21222c2 to f1d4953 Compare February 2, 2026 16:20

Fix from rebasing

a6ec6c9

emmuhamm force-pushed the emmuhamm/new-handlers branch from f1d4953 to a6ec6c9 Compare February 2, 2026 16:26

emmuhamm commented Feb 3, 2026

View reviewed changes

src/daqpytools/logging/logger.py Show resolved Hide resolved

emmuhamm added 2 commits February 3, 2026 09:43

Clean up and comment

1c03d06

Remove ERSTrace, add logging

7bcdb4b

emmuhamm changed the title ~~Emmuhamm/new handlers~~ Introduce new handlers and refactor filters Feb 3, 2026

Clean up and simplify changes

e70a6ef

emmuhamm commented Feb 4, 2026

View reviewed changes

src/daqpytools/logging/handlers.py Show resolved Hide resolved

src/daqpytools/logging/handlers.py Outdated Show resolved Hide resolved

emmuhamm commented Feb 4, 2026

View reviewed changes

src/daqpytools/logging/handlers.py Show resolved Hide resolved

emmuhamm requested a review from PawelPlesniak February 4, 2026 11:30

emmuhamm added 2 commits February 4, 2026 14:31

Clean up implementations

fed60b1

Run ruff

14aa746

emmuhamm marked this pull request as ready for review February 4, 2026 14:03

emmuhamm mentioned this pull request Feb 11, 2026

Many minor fixes to the handlers implementation #46

Merged

13 tasks

Add Lstderr

7f34870

Update formatting of the time messages

67c0171

PawelPlesniak merged commit 1956e2e into develop Feb 13, 2026
3 checks passed

PawelPlesniak deleted the emmuhamm/new-handlers branch February 13, 2026 13:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce new handlers and refactor filters#37

Introduce new handlers and refactor filters#37
PawelPlesniak merged 17 commits intodevelopfrom
emmuhamm/new-handlers

emmuhamm commented Jan 21, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

emmuhamm commented Feb 4, 2026

Uh oh!

PawelPlesniak commented Feb 13, 2026 •

edited

Loading

Uh oh!

emmuhamm commented Feb 13, 2026

Uh oh!

PawelPlesniak commented Feb 13, 2026

Uh oh!

PawelPlesniak commented Feb 13, 2026 •

edited

Loading

Uh oh!

emmuhamm commented Feb 13, 2026

Uh oh!

PawelPlesniak commented Feb 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

emmuhamm commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Added Throttling Filter (and demonstration)

Added support for Lstdout

Improves Filtering class logic and inheritance

Move away from string based log level matching to enum-based log level matching

Improves logging of the logging framework

TODO

Type of change

Testing checklist

How to test the changes

Further checks

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

emmuhamm commented Feb 4, 2026

Uh oh!

PawelPlesniak commented Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Things to check

Notes

Uh oh!

emmuhamm commented Feb 13, 2026

Uh oh!

PawelPlesniak commented Feb 13, 2026

Uh oh!

PawelPlesniak commented Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Follow up comments and question

Uh oh!

emmuhamm commented Feb 13, 2026

Uh oh!

PawelPlesniak commented Feb 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

emmuhamm commented Jan 21, 2026 •

edited

Loading

PawelPlesniak commented Feb 13, 2026 •

edited

Loading

PawelPlesniak commented Feb 13, 2026 •

edited

Loading