Add many tests and fix bugs discovered by those tests.#1139
Merged
Conversation
Introduce a `pytest.ini` file to define custom markers for integration and EPICS tests. Add multiple test files covering core functionalities, including file I/O, process run control, variable commands, and tree structure management. These additions aim to enhance test coverage and organization within the PyRogue framework, ensuring robust validation of critical components.
Update the `_RunControl.py` file to ensure that the thread join operation only occurs if the current thread is not the same as the thread being joined. This change aims to enhance thread safety and prevent potential deadlocks during run control operations.
Parse embedded config/status records through pyrogue.yamlToData() instead of calling yaml.load() directly. This keeps FileReader aligned with the rest of PyRogue's YAML handling and avoids modern PyYAML failures when no Loader is provided.
Introduce new test files for SimpleClient and Virtual interfaces, including comprehensive tests for client initialization, remote attribute handling, command execution, and listener functionality. These additions enhance test coverage and ensure robust validation of the PyRogue framework's interface components, improving overall reliability and maintainability.
Introduce a new test file for the DataReceiver class, implementing tests to verify frame processing, error handling, and connection behavior. These tests ensure accurate counting of frames and bytes, as well as proper handling of error states, enhancing the reliability and maintainability of the PyRogue framework.
Introduce a new test file for the ZmqServer class, implementing tests to verify listener registration, address resolution, operation requests, and string path handling. These tests enhance coverage and ensure the reliability of the ZmqServer component within the PyRogue framework.
Introduce a new test file for the SqlLogger class, implementing tests to verify variable and syslog row writing, handling of large integers and tuples, error handling for database connection failures, and insertion of syslog entries from a queue. These tests enhance coverage and ensure the reliability of the SqlLogger component within the PyRogue framework.
Update the SqlLogger class to unwrap JSON payloads from VariableValue objects before decoding syslog entries. This change ensures proper handling of log entries, improving the reliability of the logging functionality within the PyRogue framework.
Introduce a new test file for the Zmq integration, implementing tests to verify the interaction between a virtual client and a Zmq server. The tests cover variable access, command execution, and listener functionality, enhancing coverage and ensuring the reliability of the Zmq integration within the PyRogue framework.
Introduce a new test file for the SystemLogMonitor and VariableMonitor classes, implementing tests to verify the processing of log entries, filtering of non-system log updates, and display of variable updates. These tests enhance coverage and ensure the reliability of the monitoring components within the PyRogue framework.
Introduce a new test file for the StreamReader class, implementing tests to verify the open, close, and connection behavior of the stream reader wrapper. These tests enhance coverage and ensure the reliability of the StreamReader component within the PyRogue framework.
Introduce a new test file for the SideBandSim and MemEmulate classes, implementing tests to verify the send/receive operations, connection behavior, and error handling. These tests enhance coverage and ensure the reliability of the simulation components within the PyRogue framework.
Introduce a new test file for various model classes in the PyRogue framework, implementing tests for helper functions, unsigned and signed integer models, boolean and string models, as well as fixed-point models. These tests enhance coverage and ensure the reliability of the model components, validating their behavior across different scenarios and edge cases.
…complement handling. This improves the correctness of integer conversions in the PyRogue framework.
Introduce a new test file for the ModelVariable classes in the PyRogue framework, implementing tests for local and remote variable operations, including setting and getting values, type validation, and metadata exposure. These tests enhance coverage and ensure the reliability of the model variable components, validating their behavior across various scenarios.
Introduce a new test file for various helper functions, implementing tests for unified logging toggling, library path handling, YAML support, data formatting, function argument filtering, and documentation rendering. These tests enhance coverage and ensure the reliability of the helper components within the PyRogue framework, validating their behavior across different scenarios.
Introduce a new test file for the CPSW device and its child components in the PyRogue framework. The tests cover the export of remote variables, local variables, link variables, and commands, ensuring proper functionality and YAML file generation. These additions enhance test coverage and validate the behavior of CPSW-related components across various scenarios.
These bugs were found by the new cpsw test script.
Introduce a new test file for the StreamWriter class in the PyRogue framework, implementing tests to verify the open, close, and status behavior of the stream writer wrapper. These tests enhance coverage and ensure the reliability of the StreamWriter component, validating its interaction with configuration streams and file handling.
Introduce a new test file for block transaction helpers and local block functionality. The tests cover transaction argument verification, memory error handling, and local block set/get operations, enhancing test coverage and ensuring the reliability of block-related components within the PyRogue framework.
Updated the in-place arithmetic methods (_iadd, _isub, _imul, _imatmul, _itruediv, _ifloordiv, _imod, _ipow, _ilshift, _irshift, _iand, _ixor, _ior) to use the instance variable instead of None for setting values. This change improves the accuracy of operations and ensures that the correct variable is updated during arithmetic operations in the LocalBlock class.
Introduce a new test file for the OsCommandMemorySlave class in the PyRogue framework. The tests cover read and write operations, error handling for missing transactions, and duplicate address scenarios, enhancing test coverage and ensuring the reliability of the OsCommandMemorySlave component.
Introduce a new test file for the UdpRssiPack class, implementing tests for UDP and RSSI client interactions, variable access, and start/stop behavior. These tests enhance coverage and ensure the reliability of the UdpRssiPack component, validating its functionality across various scenarios.
…dd-tests # Conflicts: # docs/src/built_in_modules/hardware/dma/memory.rst # docs/src/built_in_modules/hardware/dma/stream.rst # docs/src/built_in_modules/hardware/raw/index.rst # docs/src/built_in_modules/protocols/batcher/index.rst # docs/src/built_in_modules/protocols/packetizer/coreV2.rst # docs/src/built_in_modules/protocols/srp/srpV0.rst # docs/src/built_in_modules/protocols/srp/srpV3.rst # docs/src/built_in_modules/protocols/uart.rst # docs/src/built_in_modules/protocols/xilinx/index.rst # docs/src/built_in_modules/utilities/fileio/index.rst # docs/src/logging/index.rst # docs/src/memory_interface/hub.rst # docs/src/memory_interface/master.rst # docs/src/memory_interface/tcp_bridge.rst # docs/src/memory_interface/transactions.rst # docs/src/pyrogue_tree/builtin_devices/prbspair.rst # docs/src/pyrogue_tree/builtin_devices/prbsrx.rst # docs/src/pyrogue_tree/builtin_devices/prbstx.rst # docs/src/pyrogue_tree/builtin_devices/stream_writer.rst # docs/src/pyrogue_tree/core/root.rst # docs/src/stream_interface/debugStreams.rst # docs/src/stream_interface/fifo.rst # docs/src/stream_interface/filter.rst # docs/src/stream_interface/tcp_bridge.rst # python/pyrogue/protocols/_Network.py # tests/test_enum.py
Introduce a new test file for the UartMemory class, implementing tests for memory read/write transactions, error handling, and worker dispatching. These tests enhance coverage and ensure the reliability of the UartMemory component, validating its behavior across various scenarios.
…ramework This commit introduces a new test file that validates the functionality of various device command variables, including local and remote variables, commands, and their interactions. The tests ensure proper handling of bulk operations, alarm states, and command execution, enhancing overall test coverage and reliability of the device command components.
This commit updates the BaseVariable class to ensure that indexed writes into ndarray-backed variables correctly handle scalar elements. It modifies the value assignment logic to convert 0-D ndarrays to scalars when necessary, improving data integrity during variable updates. Additionally, it refines index handling for single entry items to support both list and non-list index types. This bug was found by the new test harness.
…ality - Introduced tests for local variable type check errors and disabled type checking in `test_core_variables_commands.py`. - Added a test to verify data receiver functionality without type checking in `test_data_receiver.py`.
…n test_core_logging.py, including system log handling and unified logging functionality. Update FakeVariable in test_core_block_helpers.py to enable type checking.
- Introduced `BRANCH_STATUS.md` to summarize the current state of the test suite, including test layout, coverage outcomes, structural changes, and notable fixes. - Added `NEXT_STEPS.md` to outline immediate checks, high-value remaining work, and a branch wrap-up checklist. - Updated `README.md` to reference the new planning documents for better visibility on branch status and future actions.
- Updated the `test_virtual_client_cache_update_dispatch_and_monitor` to use `itertools.chain` for stable time values, ensuring consistent behavior during link checks and warning-log records.
- Updated the CI workflow to include a dedicated performance testing job, allowing for separate execution of performance tests and result collection. - Introduced a new script `ci_perf_summary.py` to format and display performance test results. - Added functionality to emit performance metrics in JSON format from various performance tests, improving result tracking and analysis. - Updated documentation to reflect new performance testing commands and usage.
- Removed redundant import statement for `emit_perf_result` in `test_variable_rate_perf.py`, streamlining the code for better readability.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR substantially expands and reorganizes the Python test suite across core PyRogue tree/runtime behavior, interfaces, protocols, file I/O, integration, and perf workloads, and includes several product fixes found by the new coverage (e.g., Process progress handling, SQL syslog decoding, UART worker queue semantics, ZMQ client logging).
Changes:
- Reorganize and add extensive pytest coverage under
tests/core,tests/interfaces,tests/protocols,tests/fileio,tests/integration, andtests/utilities, plus shared fixtures intests/conftest.py. - Split benchmark/soak workloads into
tests/perf/, add perf-result emission + a CI perf-summary workflow job. - Apply targeted product fixes in PyRogue and Rogue (e.g.,
_Process.py,_RunControl.py,_SqlLogging.py,_uart.py,ZmqClient.cpp) and update docs accordingly.
Reviewed changes
Copilot reviewed 71 out of 76 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/utilities/test_utilities_cpsw.py | New CPSW exporter tests (remote/local/link/command + YAML output). |
| tests/utilities/test_helper_functions.py | New helper-function tests (YAML helpers, wrappers, logging toggles, connect helpers, docs helpers). |
| tests/test_localvars.py | Removed legacy local-variable test (replaced by new structured tests). |
| tests/test_localcommand.py | Removed legacy local-command test (replaced by new structured tests). |
| tests/test_loadsave.py | Removed legacy load/save YAML test (replaced by new YAML/config tests). |
| tests/test_config_in.yml | Removed legacy YAML fixture (replaced by tests/core/fixtures). |
| tests/README.md | Document new test suite layout and recommended commands. |
| tests/protocols/test_protocol_uart.py | New deterministic UART protocol tests with fakes. |
| tests/protocols/test_protocol_network.py | New deterministic UDP/RSSI/packetizer wrapper tests with fakes. |
| tests/protocols/test_epics.py | Mark EPICS tests + skip when PVA context cannot be created. |
| tests/perf/test_variable_rate_perf.py | Emit perf metrics JSON per benchmark + mark as perf. |
| tests/perf/test_udp_packetizer_perf.py | Emit perf metrics JSON, improve drain timing, mark perf/integration. |
| tests/perf/test_stream_bridge_perf.py | Use dynamic ports and emit perf metrics JSON, mark perf/integration. |
| tests/perf/_perf_metrics.py | Add helper to emit perf JSON artifacts. |
| tests/NEXT_STEPS.md | Add branch handoff checklist and follow-up notes. |
| tests/interfaces/test_pydm_run_config.py | Fix module path resolution after test reorg. |
| tests/interfaces/test_pydm_rogue_plugin.py | Add Rogue PyDM plugin regression test (skips when deps missing). |
| tests/interfaces/test_os_command_memory_slave.py | Add OsCommandMemorySlave unit tests. |
| tests/interfaces/test_interfaces_zmq_server.py | Add ZmqServer unit tests with fakes. |
| tests/interfaces/test_interfaces_virtual.py | Add VirtualFactory/VirtualClient unit tests with fakes. |
| tests/interfaces/test_interfaces_virtual_integration.py | Add live VirtualClient-over-ZMQ integration tests (incl. NoServe). |
| tests/interfaces/test_interfaces_virtual_client_link.py | Fix module path resolution after test reorg. |
| tests/interfaces/test_interfaces_virtual_client_connect.py | Add VirtualClient connect/retry/cache behavior tests. |
| tests/interfaces/test_interfaces_sql_logging.py | Add SqlLogger integration/unit-style tests (SQLite) and race hardening. |
| tests/interfaces/test_interfaces_simulation.py | Add deterministic tests for simulation helpers (SideBandSim, MemEmulate, Pgp2bSim). |
| tests/interfaces/test_interfaces_simple_client.py | Add SimpleClient unit tests with fakes. |
| tests/interfaces/test_interfaces_simple_client_integration.py | Add live SimpleClient-over-ZMQ integration tests. |
| tests/interfaces/test_interfaces_monitors.py | Add tests for SystemLogMonitor and VariableMonitor behavior. |
| tests/interfaces/test_fifo.py | Add FIFO stream path test. |
| tests/interfaces/test_data_receiver.py | Add DataReceiver behavior tests (frame processing, counters, typeCheck behavior). |
| tests/integration/test_udp_packetizer_integration.py | Add end-to-end UDP/RSSI/packetizer regression test w/ out-of-order injection. |
| tests/integration/test_stream_bridge_integration.py | Add end-to-end TCP stream bridge regression test w/ dynamic port. |
| tests/integration/test_remote_enum_integration.py | Convert enum transport test to dynamic port + pytest style. |
| tests/integration/test_memory_transport_integration.py | Convert memory transport test to dynamic port + pytest asserts. |
| tests/integration/test_memory_retry_integration.py | Mark integration + refactor naming/assertions for retry emulator test. |
| tests/fileio/test_stream_writer_wrapper.py | Add StreamWriter wrapper tests with a fake backend. |
| tests/fileio/test_stream_reader_wrapper.py | Add StreamReader wrapper tests with a fake backend. |
| tests/fileio/test_fileio_smoke.py | Add (currently disabled) fileio smoke workload. |
| tests/fileio/test_core_fileio.py | Add FileReader parsing tests + StreamWriter delegation tests. |
| tests/core/test_yaml_configuration_semantics.py | Add YAML wildcard/slice/include/zip ordering semantic tests. |
| tests/core/test_poll_queue.py | Add PollQueue behavioral tests. |
| tests/core/test_model.py | Add model conversion and caching tests (UInt/Int/Bool/String/Float/Fixed). |
| tests/core/test_model_variables.py | Add local/remote variable API tests across multiple model types. |
| tests/core/test_core_variables_commands.py | Add LocalVariable/LinkVariable/LocalCommand behavior coverage. |
| tests/core/test_core_variable_wait_and_values.py | Add VariableWait + VariableValue metadata + LinkVariable edge tests. |
| tests/core/test_core_tree_root.py | Add Root lifecycle/getNode/dict/YAML/listener filter coverage. |
| tests/core/test_core_structure_edge_cases.py | Add edge-case coverage (listeners, enable propagation, resets, YAML error paths). |
| tests/core/test_core_root_device_io_edges.py | Add Root pollBlock/waitOnUpdate/log clear/YAML zip load/save + dump tests. |
| tests/core/test_core_remote_int_models.py | Refactor/rename signed remote int conversion test. |
| tests/core/test_core_process_runcontrol.py | Add Process + RunControl lifecycle/stop/error/arg handling tests. |
| tests/core/test_core_logging.py | Add root log feedback regression tests and reorder existing tests. |
| tests/core/test_core_hub_virtual_memory.py | Refactor/rename virtual hub memory test and improve assertions/comments. |
| tests/core/test_core_device_command_variable_paths.py | Add device block orchestration + remote-command helper + alarm update tests. |
| tests/core/test_core_block_overlap.py | Add overlap/block regression test coverage. |
| tests/core/test_core_block_helpers.py | Add block helper + LocalBlock behavior tests. |
| tests/core/fixtures/load_config_input.yml | Replace prior “fixture” content with actual YAML config fixture. |
| tests/conftest.py | Add shared fixtures: MemoryRoot, wait_until, dynamic port allocators, log suppression. |
| tests/BRANCH_STATUS.md | Add branch status/coverage snapshot/handoff notes. |
| tests/.gitignore | Keep *.yml ignored but allow committed tests/core/fixtures/*.yml. |
| src/rogue/interfaces/ZmqClient.cpp | Replace stdout printf timeout spam with debug logger call. |
| scripts/ci_perf_summary.py | Add script to render perf JSON results into a markdown summary. |
| python/pyrogue/protocols/_uart.py | Fix worker queue bookkeeping (task_done) to support clean joins. |
| python/pyrogue/protocols/_Network.py | Adjust link-establishment warnings during RSSI open. |
| python/pyrogue/interfaces/_SqlLogging.py | Fix syslog JSON decode path (unwrap VariableValue vs raw string). |
| python/pyrogue/_RunControl.py | Avoid joining run-control thread from within itself. |
| python/pyrogue/_Process.py | Clamp progress updates; fix argVariable set bug; adjust example loop bounds. |
| python/pyrogue/_Device.py | Clarify addRemoteVariables(..., pack=True) docstring behavior. |
| docs/src/pyrogue_tree/core/device.rst | Document packed <name>_All variable created by addRemoteVariables(..., pack=True). |
| docs/src/memory_interface/transactions.rst | Fix RST heading underline formatting. |
| docs/src/getting_started/pyrogue_tree_example/p_t_device.rst | Add docs note about packed <name>_All behavior. |
| pytest.ini | Define markers (integration, epics, perf). |
| pip_requirements.txt | Add pytest-xdist for parallel test execution. |
| .github/workflows/rogue_ci.yml | Run pytest with xdist and exclude perf by default; add dedicated perf job. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
ruck314
approved these changes
Mar 31, 2026
- Added a note on enabling logging in the batcher documentation. - Clarified the usage of `addRemoteVariables` in the PyRogue tree example, including details on indexed containers and packed `LinkVariable` entries.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This branch is a substantial Python-side test expansion, focused on raising regression coverage across the core PyRogue tree/runtime APIs, cleaning up the test suite structure, and adding higher-value end-to-end interface/integration coverage.
In addition to the original core coverage push, the branch now also:
tests/by behavior and execution styleVirtualClient/SimpleClientoverZmqServerintegration coveragetests/perf/Coverage Highlights
Current full Python run on this branch:
python -m pytest --cov171 passed, 1 skipped56%Core PyRogue modules now covered at:
pyrogue/_Root.py:91%pyrogue/_Device.py:88%pyrogue/_Node.py:88%pyrogue/_Variable.py:83%pyrogue/_Command.py:90%pyrogue/_Process.py:98%pyrogue/_HelperFunctions.py:89%pyrogue/_PollQueue.py:93%pyrogue/_Block.py:87%pyrogue/_Model.py:97%Key supporting layers now covered at:
interfaces/_SimpleClient.py:100%interfaces/_Virtual.py:86%interfaces/_ZmqServer.py:97%interfaces/_SqlLogging.py:82%interfaces/simulation.py:90%protocols/_uart.py:90%protocols/_Network.py:75%utilities/cpsw.py:88%utilities/fileio/_StreamReader.py:100%utilities/fileio/_StreamWriter.py:98%utilities/fileio/_FileReader.py:71%What This Branch Adds
At a high level, this branch now covers:
VariableValuemetadataTest Suite Reorganization
The Python test suite was reorganized into behavior-oriented directories:
tests/core/tests/interfaces/tests/fileio/tests/protocols/tests/integration/tests/perf/tests/utilities/This also included:
integration/and intocore/tests/conftest.pytests/README.mdNew High-Value Integration Coverage
VirtualClientoverZmqServerSimpleClientoverZmqServerNoServeremote-tree visibility behaviorThis makes the integration side more representative of real public API usage and less dependent on fixed local port assumptions.
Performance Test Separation
Benchmark-style workloads were split out of normal regression coverage:
tests/perf/perftests/perf/This keeps normal regression CI stable while still preserving performance tracking.
This branch is now much closer to a maintainable, behavior-oriented Python test harness, not just a pile of added test files.