Skip to content

Overhaul logging infrastructure and documentation#1132

Merged
bengineerd merged 51 commits into
pre-releasefrom
logging-infra-explore
Mar 30, 2026
Merged

Overhaul logging infrastructure and documentation#1132
bengineerd merged 51 commits into
pre-releasefrom
logging-infra-explore

Conversation

@bengineerd

@bengineerd bengineerd commented Mar 10, 2026

Copy link
Copy Markdown
Contributor

Summary
This PR expands the logging work on top of origin/pre-release in two main areas:

  1. It makes Rogue C++ logger configuration dynamic, so rogue.Logging.setFilter(...) and rogue.Logging.setLevel(...) now update existing logger instances instead of only affecting loggers created afterward.
  2. It adds optional forwarding of Rogue C++ log records into Python logging, with metadata preserved so mixed Rogue/PyRogue applications can use a single logging path.

The branch also updates a broad set of log messages, adds helper APIs for logger naming and level control from Python, adds tests for the new behavior, and refreshes the docs to match the new semantics.

Implementation Details

Dynamic updates for existing Rogue logger instances
Previously, Rogue C++ loggers effectively snapshot their level at construction time, so changing the global level or adding a prefix filter later did not reliably affect already-created objects.

This PR changes that behavior in include/rogue/Logging.h and src/rogue/Logging.cpp:

  • Logging now tracks active logger instances in a static loggers_ list.
  • Each logger stores its level in std::atomic<uint32_t> level_ instead of a plain integer.
  • A new internal helper, updateLevelLocked(), recomputes the effective level from the current global level plus prefix filters.
  • setLevel(...) now updates gblLevel_ and immediately re-applies levels to all active loggers.
  • setFilter(...) now normalizes the filter name, appends it to the filter list, and immediately re-applies levels to all active loggers.
  • Logger names are normalized through normalizeName(...), so Python-facing and Rogue-facing logger/filter naming stays consistent.

The practical result is that changing logging configuration after object construction now works for existing C++-backed Rogue objects, which is why the docs were updated to stop requiring “enable before construction” except when users specifically want constructor/startup messages.

Optional Rogue C++ -> Python log forwarding
This PR also adds an opt-in bridge from Rogue C++ logging into Python logging.

Core implementation in include/rogue/Logging.h and src/rogue/Logging.cpp:

  • Adds setForwardPython(...) / forwardPython().
  • Adds setEmitStdout(...) / emitStdout() so native stdout emission can be disabled independently.
  • intLog(...) now optionally:
    • emits to stdout via the native sink, and/or
    • acquires the GIL and forwards a Python LogRecord to logging.getLogger(name_).handle(...).
  • Forwarded records carry extra metadata fields:
    • rogue_cpp
    • rogue_tid
    • rogue_pid
    • rogue_logger
    • rogue_timestamp
    • rogue_component

This forwarding is then surfaced in PyRogue:

  • python/pyrogue/_Root.py adds unifyLogs support on Root, wires it into setUnifiedLogging(True), and copies the forwarded metadata into Root.SystemLog.
  • python/pyrogue/pydm/widgets/system_log.py is updated to display/use the richer forwarded records.
  • The branch also adds/extends helper APIs in python/pyrogue/_Node.py, including logName(...), setLogLevel(...), classLogName(), and setClassLogLevel(...), so Python code can work with the normalized logger naming model more easily.

Other notable changes

  • Adds coverage in tests/test_logging.py for the new logging behavior.
  • Refactors many Python and C++ log statements to use parameterized formatting for consistency and to avoid eager string formatting.
  • Updates docs across logging, transport, stream, memory, hardware, PRBS, file I/O, and migration pages to reflect:
    • unified logging as the preferred mixed Python/C++ path,
    • dynamic logger reconfiguration after construction,
    • common logger names and patterns,
    • forwarded C++ metadata in SystemLog.

User-facing behavior changes

  • rogue.Logging.setFilter(...) and rogue.Logging.setLevel(...) now affect existing active Rogue loggers.
  • Mixed PyRogue applications can opt into a unified logging path where Rogue C++ logs appear in Python logging handlers and Root.SystemLog.
  • Docs now clarify that logging can be enabled before or after construction; enabling early is only necessary when users want constructor/startup messages.

Testing

Suggested reviewer focus

…nality in PyRogue. Introduce class methods for Node logging, including classLogName and setClassLogLevel, to streamline logger name management. Update logging level settings for Node instances and classes, improving overall logging consistency and usability.
…tracking. Introduce std::atomic for thread-safe level updates and maintain a list of active loggers. Update logger initialization and destruction processes to ensure proper level management and logging consistency.
…abling instructions. Replace "before constructing" with "with" for clarity in setting filters for logging in AxiMemMap, MemMap, packetizer, RSSI, SRP, UDP, Xilinx, file I/O, PRBS, and memory interfaces.
… instructions for discovering logger names, setting log levels, and applying filters. Introduce helper methods for easier access to logger names and enhance examples for both Python and C++ logging practices.
…zed formatting for improved performance and consistency. This includes updates in _Command, _Device, _Node, _PollQueue, _Root, _Variable, and various interfaces, enhancing clarity and reducing the risk of formatting errors.
…for consistency and improved performance. This change aligns with recent refactoring efforts across multiple modules, enhancing clarity and reducing formatting errors.
… formatting, including detailed error messages with errno. This improves clarity and consistency in log outputs, aligning with recent refactoring efforts across the codebase.
…and response messages, enhancing clarity and consistency in log outputs.
…ized formatting, enhancing clarity and consistency in error messages. This update includes detailed context for failed message sends and bad message sizes, aligning with recent improvements across the codebase.
…ccessing C++ logger names in Python wrappers. Update examples to emphasize the use of `pyrogue.logName()` and `obj.logName()` for improved usability and consistency across logging practices.
…date documentation to clarify the new feature and its usage, including enabling forwarding and its impact on existing logging behavior. This change enhances integration between C++ and Python logging systems while maintaining backward compatibility.
… `unifyLogs` option in the Root class. This feature enables forwarding of Rogue C++ logs to Python logging while suppressing native stdout output to avoid duplicates. Update documentation to reflect the new functionality and its usage, ensuring clarity for users integrating Python and C++ logging systems.
…C++ log records in Python. Update documentation to include examples of new metadata such as `rogueCpp`, `rogueTid`, `roguePid`, `rogueLogger`, `rogueTimestamp`, and `rogueComponent`. Modify logging handlers and UI components to display this metadata, improving integration and clarity for users.
…ty and consistency. Update error messages to provide more descriptive context for binding failures and bad message sizes, enhancing overall logging quality across the codebase.
…and Python logging. Introduce `setUnifiedLogging` and `unifiedLoggingEnabled` methods for enabling/disabling logging forwarding and stdout suppression. Update `_Root` class to utilize the new logging functions, enhancing overall logging management.
…g API across various modules. Replace "Configuration API" with "Unified Logging API" and provide consistent examples for setting logger levels. This change enhances clarity and usability for users integrating Python and C++ logging systems.
… style for new Python code. Clarify the usage of `setUnifiedLogging` and provide consistent examples for integrating Python and C++ logging systems. This change enhances clarity and guides users towards the recommended logging practices.
…logger names for Rogue and PyRogue. This addition provides a quick reference for users, improving clarity on enabling logging for specific subsystems. The existing content has been streamlined to emphasize the unified logging approach and its integration with Python and C++ modules.
…r names, enhancing clarity for users on enabling logging for specific subsystems. This addition complements the existing unified logging content and improves overall usability.
…gging framework. Enhance descriptions of the `rogue.Logging` API and its relevance in older applications, standalone scripts, and debugging examples, improving user understanding of logging integration across Python and C++.
…stream and memory, hardware and utilities, and Python-only helpers. This update enhances clarity by categorizing logger names and patterns, improving user understanding of logging integration across various components.
… logging in the context of Rogue C++ logging across multiple modules. Introduce a new migration note for the unified logging style, enhancing clarity on the recommended practices for integrating Python and C++ logging systems.
…zed formatting for improved readability and consistency. This change enhances the clarity of debug messages in the LocalBlock, HelperFunctions, and UartMemory classes, aligning with the unified logging style.
…for improved readability. Add additional info logs for link establishment and UDP RX buffer configuration, aligning with the unified logging style.
…ransaction debugging. Add debug and error logs for read and write commands, ensuring consistent logging practices across the module.
…and consistency. Update log messages to use parameterized formatting, providing more detailed information during YAML config loading and link heartbeat status. This aligns with the unified logging style across the module.
…bug and warning messages. This update improves the clarity of transaction-related logs by providing additional context on timeout values, aligning with the unified logging style across the module.
…d formatting for improved consistency and readability. This update enhances the clarity of warning messages in the OpenFileBrowse, ReadNode, ReadRecursive, WriteVariable, and PyRogueLineEdit classes, aligning with the unified logging style across the module.
…rmatting for improved consistency and readability. This update enhances the clarity of warning messages in the OpenFileBrowse, ReadNode, ReadRecursive, WriteVariable, and related classes, aligning with the unified logging style across the module.
…nd consistency. Update log messages to include remote and local port information using parameterized formatting, ensuring better context for connection status and error handling. This aligns with the unified logging style across the module.
# Conflicts:
#	docs/src/logging/index.rst
Consolidate API reference entries for Python and C++ into single lines for better readability. Update section titles from "Related Docs" to "Related Topics" to enhance clarity. These changes aim to streamline the documentation structure and improve user navigation.
@bengineerd bengineerd changed the base branch from doc-update-6 to pre-release March 14, 2026 03:42
Comment thread include/rogue/protocols/xilinx/XvcConnection.h
…dation

Introduce a new test suite in `test_logging.py` that includes tests for logging name and level helpers, unified logging behavior, and the capturing of metadata in system logs. This addition aims to ensure robust validation of logging features within the PyRogue framework, improving overall reliability and maintainability of the logging system.
@ruck314 ruck314 requested a review from slacrherbst March 16, 2026 18:24
@bengineerd bengineerd changed the base branch from pre-release to doc-update-6 March 16, 2026 18:24
…explore

# Conflicts:
#	docs/src/built_in_modules/hardware/dma/memory.rst
#	docs/src/built_in_modules/hardware/dma/stream.rst
#	docs/src/built_in_modules/hardware/raw/index.rst
#	docs/src/built_in_modules/protocols/batcher/index.rst
#	docs/src/built_in_modules/protocols/packetizer/coreV2.rst
#	docs/src/built_in_modules/protocols/srp/srpV0.rst
#	docs/src/built_in_modules/protocols/srp/srpV3.rst
#	docs/src/built_in_modules/protocols/uart.rst
#	docs/src/built_in_modules/protocols/xilinx/index.rst
#	docs/src/built_in_modules/utilities/fileio/index.rst
#	docs/src/logging/index.rst
#	docs/src/memory_interface/hub.rst
#	docs/src/memory_interface/master.rst
#	docs/src/memory_interface/tcp_bridge.rst
#	docs/src/memory_interface/transactions.rst
#	docs/src/pyrogue_tree/builtin_devices/prbspair.rst
#	docs/src/pyrogue_tree/builtin_devices/prbsrx.rst
#	docs/src/pyrogue_tree/builtin_devices/prbstx.rst
#	docs/src/pyrogue_tree/builtin_devices/stream_writer.rst
#	docs/src/pyrogue_tree/core/root.rst
#	docs/src/stream_interface/debugStreams.rst
#	docs/src/stream_interface/fifo.rst
#	docs/src/stream_interface/filter.rst
#	docs/src/stream_interface/tcp_bridge.rst
#	python/pyrogue/protocols/_Network.py
@bengineerd bengineerd changed the base branch from doc-update-6 to pre-release March 16, 2026 20:40
…gging can be enabled before or after construction. Emphasize that enabling logging before construction is only necessary for capturing constructor or early startup messages.
@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 40.43478% with 137 lines in your changes missing coverage. Please review.
✅ Project coverage is 29.91%. Comparing base (455967d) to head (58726b4).
⚠️ Report is 1068 commits behind head on pre-release.

Files with missing lines Patch % Lines
python/pyrogue/pydm/widgets/system_log.py 0.00% 34 Missing ⚠️
python/pyrogue/interfaces/__init__.py 0.00% 17 Missing ⚠️
python/pyrogue/_Variable.py 18.75% 13 Missing ⚠️
python/pyrogue/_Node.py 76.00% 7 Missing and 5 partials ⚠️
python/pyrogue/interfaces/_OsCommandMemorySlave.py 12.50% 7 Missing ⚠️
python/pyrogue/_Root.py 82.85% 6 Missing ⚠️
python/pyrogue/interfaces/simulation.py 0.00% 6 Missing ⚠️
python/pyrogue/protocols/_uart.py 0.00% 6 Missing ⚠️
python/pyrogue/interfaces/_SqlLogging.py 0.00% 5 Missing ⚠️
python/pyrogue/utilities/fileio/_FileReader.py 0.00% 5 Missing ⚠️
... and 13 more
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@               Coverage Diff               @@
##           pre-release    #1132      +/-   ##
===============================================
+ Coverage        29.54%   29.91%   +0.37%     
===============================================
  Files               65       68       +3     
  Lines             6644     7123     +479     
  Branches          1015     1008       -7     
===============================================
+ Hits              1963     2131     +168     
- Misses            4527     4834     +307     
- Partials           154      158       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@bengineerd bengineerd marked this pull request as ready for review March 18, 2026 21:10
…logger names and patterns. Update index.rst to include new guidelines for configuring logging in PyRogue applications and clarify message emission identification for older code. Replace the previous logger-name table with a dedicated logger_names.rst for better organization.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR overhauls Rogue/PyRogue logging so C++ logger configuration updates apply to already-created logger instances and (optionally) forwards Rogue C++ log records into Python logging to support a unified logging pipeline, with docs/tests updated to match.

Changes:

  • Update Rogue C++ rogue::Logging to normalize names, track active instances, dynamically recompute levels, and optionally forward C++ logs into Python logging (with extra metadata fields).
  • Add/extend PyRogue helper APIs for unified logging enablement and logger naming/level control, plus new tests for the new behaviors.
  • Refresh many log messages and documentation pages to reflect unified logging, dynamic reconfiguration semantics, and logger naming patterns.

Reviewed changes

Copilot reviewed 78 out of 78 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
tests/test_logging.py Adds tests for logger naming helpers, unified logging toggles, forwarding behavior, and SystemLog metadata capture.
src/rogue/protocols/xilinx/Xvc.cpp Fixes/modernizes debug log formatting for frame sizes.
src/rogue/protocols/udp/Server.cpp Improves UDP server operational logging and peer-address visibility.
src/rogue/protocols/udp/Client.cpp Improves UDP client operational logging and error context.
src/rogue/protocols/rssi/Controller.cpp Refines RSSI log messages and adjusts severity for “open” state logs.
src/rogue/protocols/packetizer/ControllerV2.cpp Clarifies empty-frame warnings.
src/rogue/protocols/packetizer/ControllerV1.cpp Clarifies empty-frame warnings.
src/rogue/interfaces/stream/TcpCore.cpp Adds lifecycle logs and improves ZMQ error reporting.
src/rogue/interfaces/memory/Transaction.cpp Adds timeout context to transaction timeout/refresh warnings.
src/rogue/interfaces/memory/TcpServer.cpp Adds lifecycle logs and improves “bad message sizes” diagnostics.
src/rogue/interfaces/memory/TcpClient.cpp Improves send failure context and “bad message sizes” diagnostics.
src/rogue/interfaces/ZmqServer.cpp Improves bind-failure diagnostics and adjusts server startup logging.
src/rogue/Logging.cpp Implements dynamic level propagation, name normalization, and optional Python forwarding/stdout control.
include/rogue/Logging.h Adds APIs/state for forwarding/stdout control, name normalization, and atomic per-logger levels.
python/pyrogue/utilities/fileio/_FileReader.py Converts eager f-string logging to parameterized logging.
python/pyrogue/pydm/widgets/system_log.py Displays forwarded Rogue C++ metadata and adds tag stamps in the UI.
python/pyrogue/pydm/widgets/line_edit.py Converts error logging to parameterized formatting.
python/pyrogue/pydm/tools/write_recursive_tool.py Converts warning logging to parameterized formatting.
python/pyrogue/pydm/tools/write_node_tool.py Converts warning logging to parameterized formatting.
python/pyrogue/pydm/tools/read_recursive_tool.py Converts warning logging to parameterized formatting.
python/pyrogue/pydm/tools/read_node_tool.py Converts warning logging to parameterized formatting.
python/pyrogue/pydm/tools/generic_file_tool.py Converts warning logging to parameterized formatting.
python/pyrogue/pydm/init.py Defers heavy PyDM imports to runtime and improves typing ergonomics.
python/pyrogue/protocols/gpib.py Converts eager f-string logging to parameterized logging.
python/pyrogue/protocols/epicsV4.py Converts eager f-string logging to parameterized logging.
python/pyrogue/protocols/_uart.py Improves UART transaction debug logging and converts commented logs into active parameterized logs.
python/pyrogue/protocols/_Network.py Improves network bring-up logging and adds negotiated RX-buffer configuration logs.
python/pyrogue/interfaces/simulation.py Converts eager f-string logging to parameterized logging and fixes error-string formatting.
python/pyrogue/interfaces/init.py Enhances SystemLogMonitor output to include forwarded Rogue C++ metadata.
python/pyrogue/interfaces/_SqlLogging.py Converts eager f-string logging to parameterized logging.
python/pyrogue/interfaces/_OsCommandMemorySlave.py Adds a logger and richer debug/error messages around command dispatch.
python/pyrogue/_Variable.py Fixes logging format issues and converts to parameterized logging.
python/pyrogue/_Root.py Adds unifyLogs option and captures forwarded Rogue metadata into SystemLog.
python/pyrogue/_PollQueue.py Converts eager f-string logging to parameterized logging.
python/pyrogue/_Node.py Adds logger naming and log-level helper APIs (Python + Rogue).
python/pyrogue/_HelperFunctions.py Adds unified logging enable/disable helpers (setUnifiedLogging, unifiedLoggingEnabled).
python/pyrogue/_Device.py Improves lifecycle/management logging and converts to parameterized logging.
python/pyrogue/_Command.py Converts eager formatting to parameterized logging.
docs/src/stream_interface/tcp_bridge.rst Documents unified vs legacy logging APIs and dynamic reconfig semantics.
docs/src/stream_interface/rate_drop.rst Updates logging guidance to reflect dynamic reconfiguration.
docs/src/stream_interface/filter.rst Documents logger name and unified/legacy configuration snippets.
docs/src/stream_interface/fifo.rst Documents logger name and unified/legacy configuration snippets.
docs/src/stream_interface/debugStreams.rst Documents dynamic logger pattern and unified/legacy configuration.
docs/src/pyrogue_tree/core/root.rst Documents unifyLogs and forwarded C++ metadata in SystemLog.
docs/src/pyrogue_tree/core/block.rst Adds unified/legacy logging configuration examples for block logging.
docs/src/pyrogue_tree/client_interfaces/zmq_server.rst Updates transport logging guidance for unified logging.
docs/src/pyrogue_tree/client_interfaces/virtual.rst Minor logging wording update.
docs/src/migration/logging_unified.rst New migration guide for unified logging usage.
docs/src/migration/index.rst Adds unified logging migration page to the migration toctree.
docs/src/memory_interface/transactions.rst Adds transaction logging section and logger names/APIs.
docs/src/memory_interface/tcp_bridge.rst Documents unified/legacy logging configuration for TCP memory bridge.
docs/src/memory_interface/master.rst Documents logger name and unified/legacy configuration snippets.
docs/src/memory_interface/hub.rst Documents logger name and unified/legacy configuration snippets.
docs/src/logging/logger_names.rst New consolidated logger-name reference page.
docs/src/logging/index.rst Reframes docs around unified logging path and adds recipes/examples.
docs/src/installing/miniforge_build.rst Updates build command example to parallel build invocation.
docs/src/built_in_modules/utilities/prbs/index.rst Adds unified/legacy logging configuration snippet.
docs/src/built_in_modules/utilities/fileio/writing.rst Adds unified/legacy logging configuration snippet.
docs/src/built_in_modules/utilities/fileio/python_reader.rst Minor logging wording update.
docs/src/built_in_modules/utilities/fileio/index.rst Minor formatting adjustment.
docs/src/built_in_modules/simulation/pgp2b.rst Updates logging guidance to include unified/legacy API examples.
docs/src/built_in_modules/protocols/xilinx/index.rst Adds unified/legacy logging configuration snippet.
docs/src/built_in_modules/protocols/udp/index.rst Adds unified/legacy logging configuration snippet.
docs/src/built_in_modules/protocols/uart.rst Adds explicit UART logger naming pattern examples.
docs/src/built_in_modules/protocols/srp/srpV3.rst Adds unified/legacy logging configuration snippet and updates semantics.
docs/src/built_in_modules/protocols/srp/srpV0.rst Adds unified/legacy logging configuration snippet and updates semantics.
docs/src/built_in_modules/protocols/rssi/index.rst Adds unified/legacy logging configuration snippet.
docs/src/built_in_modules/protocols/packetizer/coreV2.rst Adds unified/legacy logging configuration snippet and updates semantics.
docs/src/built_in_modules/protocols/packetizer/core.rst Adds unified/legacy logging configuration snippet.
docs/src/built_in_modules/protocols/network.rst Adds unified/legacy logging configuration guidance.
docs/src/built_in_modules/protocols/gpib.rst Adds concise logging configuration snippet.
docs/src/built_in_modules/protocols/epicsV4/epicspvserver.rst Minor logging wording update.
docs/src/built_in_modules/protocols/epicsV4/epicspvholder.rst Minor logging wording update.
docs/src/built_in_modules/protocols/batcher/index.rst Adds unified/legacy logging configuration snippet and updates semantics.
docs/src/built_in_modules/interfaces/sql.rst Minor logging wording update.
docs/src/built_in_modules/hardware/raw/index.rst Adds unified/legacy logging configuration snippet and updates semantics.
docs/src/built_in_modules/hardware/dma/stream.rst Adds unified/legacy logging configuration snippet and updates semantics.
docs/src/built_in_modules/hardware/dma/memory.rst Adds unified/legacy logging configuration snippet and updates semantics.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/rogue/Logging.cpp
Comment thread docs/src/installing/miniforge_build.rst
Comment thread python/pyrogue/_Node.py Outdated
Comment thread src/rogue/Logging.cpp Outdated
Comment thread src/rogue/Logging.cpp Outdated
ruck314 and others added 4 commits March 30, 2026 11:19
…rror handling for Python logger and implemented level filtering to prevent unnecessary log forwarding. Introduced tests for unified logging behavior, including exception handling in custom logging handlers.
Introduced a new _Logging.py module containing logging utility functions for PyRogue, including logger initialization, exception logging, and log level management. Updated _Node.py to utilize these new logging functions, enhancing the logging capabilities across the PyRogue framework.
@bengineerd bengineerd merged commit 553d8a4 into pre-release Mar 30, 2026
6 checks passed
@bengineerd bengineerd deleted the logging-infra-explore branch March 30, 2026 22:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants