Overhaul logging infrastructure and documentation#1132
Conversation
…nality in PyRogue. Introduce class methods for Node logging, including classLogName and setClassLogLevel, to streamline logger name management. Update logging level settings for Node instances and classes, improving overall logging consistency and usability.
…tracking. Introduce std::atomic for thread-safe level updates and maintain a list of active loggers. Update logger initialization and destruction processes to ensure proper level management and logging consistency.
…abling instructions. Replace "before constructing" with "with" for clarity in setting filters for logging in AxiMemMap, MemMap, packetizer, RSSI, SRP, UDP, Xilinx, file I/O, PRBS, and memory interfaces.
… instructions for discovering logger names, setting log levels, and applying filters. Introduce helper methods for easier access to logger names and enhance examples for both Python and C++ logging practices.
…zed formatting for improved performance and consistency. This includes updates in _Command, _Device, _Node, _PollQueue, _Root, _Variable, and various interfaces, enhancing clarity and reducing the risk of formatting errors.
…for consistency and improved performance. This change aligns with recent refactoring efforts across multiple modules, enhancing clarity and reducing formatting errors.
… formatting, including detailed error messages with errno. This improves clarity and consistency in log outputs, aligning with recent refactoring efforts across the codebase.
…and response messages, enhancing clarity and consistency in log outputs.
…ized formatting, enhancing clarity and consistency in error messages. This update includes detailed context for failed message sends and bad message sizes, aligning with recent improvements across the codebase.
…ccessing C++ logger names in Python wrappers. Update examples to emphasize the use of `pyrogue.logName()` and `obj.logName()` for improved usability and consistency across logging practices.
…date documentation to clarify the new feature and its usage, including enabling forwarding and its impact on existing logging behavior. This change enhances integration between C++ and Python logging systems while maintaining backward compatibility.
… `unifyLogs` option in the Root class. This feature enables forwarding of Rogue C++ logs to Python logging while suppressing native stdout output to avoid duplicates. Update documentation to reflect the new functionality and its usage, ensuring clarity for users integrating Python and C++ logging systems.
…C++ log records in Python. Update documentation to include examples of new metadata such as `rogueCpp`, `rogueTid`, `roguePid`, `rogueLogger`, `rogueTimestamp`, and `rogueComponent`. Modify logging handlers and UI components to display this metadata, improving integration and clarity for users.
…ty and consistency. Update error messages to provide more descriptive context for binding failures and bad message sizes, enhancing overall logging quality across the codebase.
…and Python logging. Introduce `setUnifiedLogging` and `unifiedLoggingEnabled` methods for enabling/disabling logging forwarding and stdout suppression. Update `_Root` class to utilize the new logging functions, enhancing overall logging management.
…g API across various modules. Replace "Configuration API" with "Unified Logging API" and provide consistent examples for setting logger levels. This change enhances clarity and usability for users integrating Python and C++ logging systems.
… style for new Python code. Clarify the usage of `setUnifiedLogging` and provide consistent examples for integrating Python and C++ logging systems. This change enhances clarity and guides users towards the recommended logging practices.
…logger names for Rogue and PyRogue. This addition provides a quick reference for users, improving clarity on enabling logging for specific subsystems. The existing content has been streamlined to emphasize the unified logging approach and its integration with Python and C++ modules.
…r names, enhancing clarity for users on enabling logging for specific subsystems. This addition complements the existing unified logging content and improves overall usability.
…gging framework. Enhance descriptions of the `rogue.Logging` API and its relevance in older applications, standalone scripts, and debugging examples, improving user understanding of logging integration across Python and C++.
…stream and memory, hardware and utilities, and Python-only helpers. This update enhances clarity by categorizing logger names and patterns, improving user understanding of logging integration across various components.
… logging in the context of Rogue C++ logging across multiple modules. Introduce a new migration note for the unified logging style, enhancing clarity on the recommended practices for integrating Python and C++ logging systems.
…zed formatting for improved readability and consistency. This change enhances the clarity of debug messages in the LocalBlock, HelperFunctions, and UartMemory classes, aligning with the unified logging style.
…for improved readability. Add additional info logs for link establishment and UDP RX buffer configuration, aligning with the unified logging style.
…ransaction debugging. Add debug and error logs for read and write commands, ensuring consistent logging practices across the module.
…and consistency. Update log messages to use parameterized formatting, providing more detailed information during YAML config loading and link heartbeat status. This aligns with the unified logging style across the module.
…bug and warning messages. This update improves the clarity of transaction-related logs by providing additional context on timeout values, aligning with the unified logging style across the module.
…d formatting for improved consistency and readability. This update enhances the clarity of warning messages in the OpenFileBrowse, ReadNode, ReadRecursive, WriteVariable, and PyRogueLineEdit classes, aligning with the unified logging style across the module.
…rmatting for improved consistency and readability. This update enhances the clarity of warning messages in the OpenFileBrowse, ReadNode, ReadRecursive, WriteVariable, and related classes, aligning with the unified logging style across the module.
…nd consistency. Update log messages to include remote and local port information using parameterized formatting, ensuring better context for connection status and error handling. This aligns with the unified logging style across the module.
# Conflicts: # docs/src/logging/index.rst
Consolidate API reference entries for Python and C++ into single lines for better readability. Update section titles from "Related Docs" to "Related Topics" to enhance clarity. These changes aim to streamline the documentation structure and improve user navigation.
…dation Introduce a new test suite in `test_logging.py` that includes tests for logging name and level helpers, unified logging behavior, and the capturing of metadata in system logs. This addition aims to ensure robust validation of logging features within the PyRogue framework, improving overall reliability and maintainability of the logging system.
…explore # Conflicts: # docs/src/built_in_modules/hardware/dma/memory.rst # docs/src/built_in_modules/hardware/dma/stream.rst # docs/src/built_in_modules/hardware/raw/index.rst # docs/src/built_in_modules/protocols/batcher/index.rst # docs/src/built_in_modules/protocols/packetizer/coreV2.rst # docs/src/built_in_modules/protocols/srp/srpV0.rst # docs/src/built_in_modules/protocols/srp/srpV3.rst # docs/src/built_in_modules/protocols/uart.rst # docs/src/built_in_modules/protocols/xilinx/index.rst # docs/src/built_in_modules/utilities/fileio/index.rst # docs/src/logging/index.rst # docs/src/memory_interface/hub.rst # docs/src/memory_interface/master.rst # docs/src/memory_interface/tcp_bridge.rst # docs/src/memory_interface/transactions.rst # docs/src/pyrogue_tree/builtin_devices/prbspair.rst # docs/src/pyrogue_tree/builtin_devices/prbsrx.rst # docs/src/pyrogue_tree/builtin_devices/prbstx.rst # docs/src/pyrogue_tree/builtin_devices/stream_writer.rst # docs/src/pyrogue_tree/core/root.rst # docs/src/stream_interface/debugStreams.rst # docs/src/stream_interface/fifo.rst # docs/src/stream_interface/filter.rst # docs/src/stream_interface/tcp_bridge.rst # python/pyrogue/protocols/_Network.py
…gging can be enabled before or after construction. Emphasize that enabling logging before construction is only necessary for capturing constructor or early startup messages.
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## pre-release #1132 +/- ##
===============================================
+ Coverage 29.54% 29.91% +0.37%
===============================================
Files 65 68 +3
Lines 6644 7123 +479
Branches 1015 1008 -7
===============================================
+ Hits 1963 2131 +168
- Misses 4527 4834 +307
- Partials 154 158 +4 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
…logger names and patterns. Update index.rst to include new guidelines for configuring logging in PyRogue applications and clarify message emission identification for older code. Replace the previous logger-name table with a dedicated logger_names.rst for better organization.
…tion. Added future annotations and conditional imports for improved type hinting during static analysis.
There was a problem hiding this comment.
Pull request overview
This PR overhauls Rogue/PyRogue logging so C++ logger configuration updates apply to already-created logger instances and (optionally) forwards Rogue C++ log records into Python logging to support a unified logging pipeline, with docs/tests updated to match.
Changes:
- Update Rogue C++
rogue::Loggingto normalize names, track active instances, dynamically recompute levels, and optionally forward C++ logs into Pythonlogging(with extra metadata fields). - Add/extend PyRogue helper APIs for unified logging enablement and logger naming/level control, plus new tests for the new behaviors.
- Refresh many log messages and documentation pages to reflect unified logging, dynamic reconfiguration semantics, and logger naming patterns.
Reviewed changes
Copilot reviewed 78 out of 78 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_logging.py | Adds tests for logger naming helpers, unified logging toggles, forwarding behavior, and SystemLog metadata capture. |
| src/rogue/protocols/xilinx/Xvc.cpp | Fixes/modernizes debug log formatting for frame sizes. |
| src/rogue/protocols/udp/Server.cpp | Improves UDP server operational logging and peer-address visibility. |
| src/rogue/protocols/udp/Client.cpp | Improves UDP client operational logging and error context. |
| src/rogue/protocols/rssi/Controller.cpp | Refines RSSI log messages and adjusts severity for “open” state logs. |
| src/rogue/protocols/packetizer/ControllerV2.cpp | Clarifies empty-frame warnings. |
| src/rogue/protocols/packetizer/ControllerV1.cpp | Clarifies empty-frame warnings. |
| src/rogue/interfaces/stream/TcpCore.cpp | Adds lifecycle logs and improves ZMQ error reporting. |
| src/rogue/interfaces/memory/Transaction.cpp | Adds timeout context to transaction timeout/refresh warnings. |
| src/rogue/interfaces/memory/TcpServer.cpp | Adds lifecycle logs and improves “bad message sizes” diagnostics. |
| src/rogue/interfaces/memory/TcpClient.cpp | Improves send failure context and “bad message sizes” diagnostics. |
| src/rogue/interfaces/ZmqServer.cpp | Improves bind-failure diagnostics and adjusts server startup logging. |
| src/rogue/Logging.cpp | Implements dynamic level propagation, name normalization, and optional Python forwarding/stdout control. |
| include/rogue/Logging.h | Adds APIs/state for forwarding/stdout control, name normalization, and atomic per-logger levels. |
| python/pyrogue/utilities/fileio/_FileReader.py | Converts eager f-string logging to parameterized logging. |
| python/pyrogue/pydm/widgets/system_log.py | Displays forwarded Rogue C++ metadata and adds tag stamps in the UI. |
| python/pyrogue/pydm/widgets/line_edit.py | Converts error logging to parameterized formatting. |
| python/pyrogue/pydm/tools/write_recursive_tool.py | Converts warning logging to parameterized formatting. |
| python/pyrogue/pydm/tools/write_node_tool.py | Converts warning logging to parameterized formatting. |
| python/pyrogue/pydm/tools/read_recursive_tool.py | Converts warning logging to parameterized formatting. |
| python/pyrogue/pydm/tools/read_node_tool.py | Converts warning logging to parameterized formatting. |
| python/pyrogue/pydm/tools/generic_file_tool.py | Converts warning logging to parameterized formatting. |
| python/pyrogue/pydm/init.py | Defers heavy PyDM imports to runtime and improves typing ergonomics. |
| python/pyrogue/protocols/gpib.py | Converts eager f-string logging to parameterized logging. |
| python/pyrogue/protocols/epicsV4.py | Converts eager f-string logging to parameterized logging. |
| python/pyrogue/protocols/_uart.py | Improves UART transaction debug logging and converts commented logs into active parameterized logs. |
| python/pyrogue/protocols/_Network.py | Improves network bring-up logging and adds negotiated RX-buffer configuration logs. |
| python/pyrogue/interfaces/simulation.py | Converts eager f-string logging to parameterized logging and fixes error-string formatting. |
| python/pyrogue/interfaces/init.py | Enhances SystemLogMonitor output to include forwarded Rogue C++ metadata. |
| python/pyrogue/interfaces/_SqlLogging.py | Converts eager f-string logging to parameterized logging. |
| python/pyrogue/interfaces/_OsCommandMemorySlave.py | Adds a logger and richer debug/error messages around command dispatch. |
| python/pyrogue/_Variable.py | Fixes logging format issues and converts to parameterized logging. |
| python/pyrogue/_Root.py | Adds unifyLogs option and captures forwarded Rogue metadata into SystemLog. |
| python/pyrogue/_PollQueue.py | Converts eager f-string logging to parameterized logging. |
| python/pyrogue/_Node.py | Adds logger naming and log-level helper APIs (Python + Rogue). |
| python/pyrogue/_HelperFunctions.py | Adds unified logging enable/disable helpers (setUnifiedLogging, unifiedLoggingEnabled). |
| python/pyrogue/_Device.py | Improves lifecycle/management logging and converts to parameterized logging. |
| python/pyrogue/_Command.py | Converts eager formatting to parameterized logging. |
| docs/src/stream_interface/tcp_bridge.rst | Documents unified vs legacy logging APIs and dynamic reconfig semantics. |
| docs/src/stream_interface/rate_drop.rst | Updates logging guidance to reflect dynamic reconfiguration. |
| docs/src/stream_interface/filter.rst | Documents logger name and unified/legacy configuration snippets. |
| docs/src/stream_interface/fifo.rst | Documents logger name and unified/legacy configuration snippets. |
| docs/src/stream_interface/debugStreams.rst | Documents dynamic logger pattern and unified/legacy configuration. |
| docs/src/pyrogue_tree/core/root.rst | Documents unifyLogs and forwarded C++ metadata in SystemLog. |
| docs/src/pyrogue_tree/core/block.rst | Adds unified/legacy logging configuration examples for block logging. |
| docs/src/pyrogue_tree/client_interfaces/zmq_server.rst | Updates transport logging guidance for unified logging. |
| docs/src/pyrogue_tree/client_interfaces/virtual.rst | Minor logging wording update. |
| docs/src/migration/logging_unified.rst | New migration guide for unified logging usage. |
| docs/src/migration/index.rst | Adds unified logging migration page to the migration toctree. |
| docs/src/memory_interface/transactions.rst | Adds transaction logging section and logger names/APIs. |
| docs/src/memory_interface/tcp_bridge.rst | Documents unified/legacy logging configuration for TCP memory bridge. |
| docs/src/memory_interface/master.rst | Documents logger name and unified/legacy configuration snippets. |
| docs/src/memory_interface/hub.rst | Documents logger name and unified/legacy configuration snippets. |
| docs/src/logging/logger_names.rst | New consolidated logger-name reference page. |
| docs/src/logging/index.rst | Reframes docs around unified logging path and adds recipes/examples. |
| docs/src/installing/miniforge_build.rst | Updates build command example to parallel build invocation. |
| docs/src/built_in_modules/utilities/prbs/index.rst | Adds unified/legacy logging configuration snippet. |
| docs/src/built_in_modules/utilities/fileio/writing.rst | Adds unified/legacy logging configuration snippet. |
| docs/src/built_in_modules/utilities/fileio/python_reader.rst | Minor logging wording update. |
| docs/src/built_in_modules/utilities/fileio/index.rst | Minor formatting adjustment. |
| docs/src/built_in_modules/simulation/pgp2b.rst | Updates logging guidance to include unified/legacy API examples. |
| docs/src/built_in_modules/protocols/xilinx/index.rst | Adds unified/legacy logging configuration snippet. |
| docs/src/built_in_modules/protocols/udp/index.rst | Adds unified/legacy logging configuration snippet. |
| docs/src/built_in_modules/protocols/uart.rst | Adds explicit UART logger naming pattern examples. |
| docs/src/built_in_modules/protocols/srp/srpV3.rst | Adds unified/legacy logging configuration snippet and updates semantics. |
| docs/src/built_in_modules/protocols/srp/srpV0.rst | Adds unified/legacy logging configuration snippet and updates semantics. |
| docs/src/built_in_modules/protocols/rssi/index.rst | Adds unified/legacy logging configuration snippet. |
| docs/src/built_in_modules/protocols/packetizer/coreV2.rst | Adds unified/legacy logging configuration snippet and updates semantics. |
| docs/src/built_in_modules/protocols/packetizer/core.rst | Adds unified/legacy logging configuration snippet. |
| docs/src/built_in_modules/protocols/network.rst | Adds unified/legacy logging configuration guidance. |
| docs/src/built_in_modules/protocols/gpib.rst | Adds concise logging configuration snippet. |
| docs/src/built_in_modules/protocols/epicsV4/epicspvserver.rst | Minor logging wording update. |
| docs/src/built_in_modules/protocols/epicsV4/epicspvholder.rst | Minor logging wording update. |
| docs/src/built_in_modules/protocols/batcher/index.rst | Adds unified/legacy logging configuration snippet and updates semantics. |
| docs/src/built_in_modules/interfaces/sql.rst | Minor logging wording update. |
| docs/src/built_in_modules/hardware/raw/index.rst | Adds unified/legacy logging configuration snippet and updates semantics. |
| docs/src/built_in_modules/hardware/dma/stream.rst | Adds unified/legacy logging configuration snippet and updates semantics. |
| docs/src/built_in_modules/hardware/dma/memory.rst | Adds unified/legacy logging configuration snippet and updates semantics. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…rror handling for Python logger and implemented level filtering to prevent unnecessary log forwarding. Introduced tests for unified logging behavior, including exception handling in custom logging handlers.
Introduced a new _Logging.py module containing logging utility functions for PyRogue, including logger initialization, exception logging, and log level management. Updated _Node.py to utilize these new logging functions, enhancing the logging capabilities across the PyRogue framework.
Summary
This PR expands the logging work on top of
origin/pre-releasein two main areas:rogue.Logging.setFilter(...)androgue.Logging.setLevel(...)now update existing logger instances instead of only affecting loggers created afterward.logging, with metadata preserved so mixed Rogue/PyRogue applications can use a single logging path.The branch also updates a broad set of log messages, adds helper APIs for logger naming and level control from Python, adds tests for the new behavior, and refreshes the docs to match the new semantics.
Implementation Details
Dynamic updates for existing Rogue logger instances
Previously, Rogue C++ loggers effectively snapshot their level at construction time, so changing the global level or adding a prefix filter later did not reliably affect already-created objects.
This PR changes that behavior in include/rogue/Logging.h and src/rogue/Logging.cpp:
Loggingnow tracks active logger instances in a staticloggers_list.std::atomic<uint32_t> level_instead of a plain integer.updateLevelLocked(), recomputes the effective level from the current global level plus prefix filters.setLevel(...)now updatesgblLevel_and immediately re-applies levels to all active loggers.setFilter(...)now normalizes the filter name, appends it to the filter list, and immediately re-applies levels to all active loggers.normalizeName(...), so Python-facing and Rogue-facing logger/filter naming stays consistent.The practical result is that changing logging configuration after object construction now works for existing C++-backed Rogue objects, which is why the docs were updated to stop requiring “enable before construction” except when users specifically want constructor/startup messages.
Optional Rogue C++ -> Python log forwarding
This PR also adds an opt-in bridge from Rogue C++ logging into Python
logging.Core implementation in include/rogue/Logging.h and src/rogue/Logging.cpp:
setForwardPython(...)/forwardPython().setEmitStdout(...)/emitStdout()so native stdout emission can be disabled independently.intLog(...)now optionally:LogRecordtologging.getLogger(name_).handle(...).rogue_cpprogue_tidrogue_pidrogue_loggerrogue_timestamprogue_componentThis forwarding is then surfaced in PyRogue:
unifyLogssupport onRoot, wires it intosetUnifiedLogging(True), and copies the forwarded metadata intoRoot.SystemLog.logName(...),setLogLevel(...),classLogName(), andsetClassLogLevel(...), so Python code can work with the normalized logger naming model more easily.Other notable changes
SystemLog.User-facing behavior changes
rogue.Logging.setFilter(...)androgue.Logging.setLevel(...)now affect existing active Rogue loggers.Root.SystemLog.Testing
Suggested reviewer focus