Skip to content

Revamp Rogue documentation structure, narrative guides, and built-in module coverage#1131

Merged
bengineerd merged 171 commits into
pre-releasefrom
doc-update-6
Mar 16, 2026
Merged

Revamp Rogue documentation structure, narrative guides, and built-in module coverage#1131
bengineerd merged 171 commits into
pre-releasefrom
doc-update-6

Conversation

@bengineerd

@bengineerd bengineerd commented Mar 9, 2026

Copy link
Copy Markdown
Contributor

Summary
This branch is a large documentation rework that reorganizes Rogue around clearer top-level concepts, replaces the older interface-driven structure with better narrative sections, expands practical user guidance across PyRogue, stream, memory, built-in modules, and protocols, and significantly improves the generated API reference experience.

It also moves a large amount of content into new locations, rewrites many thin or outdated pages into real narrative documentation, refreshes navigation so the docs behave more like a coherent manual instead of a collection of disconnected reference stubs, and adds new docs tooling for generated API pages and local source-code browsing.

What changed

Documentation architecture

  • Established top-level sections and reduced legacy navigation clutter
  • Reworked the docs entry point and sidebar organization
  • Reframed the docs around user-facing concepts instead of older implementation-oriented structure

Top-level documentation structure
The docs now center on user-facing concepts instead of the old interfaces/-centric layout:

  • Introduction
  • Installing & Compiling Rogue
  • PyRogue Tree
  • Stream Interface
  • Memory Interface
  • Built-in Modules
  • Logging In Rogue
  • Using the PyDM Gui
  • Tutorials
  • Cookbook
  • Migration Notes
  • API Reference

Earlier quick-start material was folded into the installation/tutorial flow instead of remaining a standalone primary section.

Core narrative sections

  • Reorganized and expanded pyrogue_tree into clearer core concepts such as root, device, variable, command, block, model, poll queue, and YAML configuration
  • Reworked stream_interface into a coherent narrative section covering connecting, sending, receiving, frame model, TCP bridge, and built-in helpers like FIFO, Filter, and RateDrop
  • Reworked memory_interface into clearer conceptual docs for connecting, master/slave behavior, hub usage, transactions, and TCP bridging
  • Added stronger lifecycle, bulk-operation, transport, and integration guidance across these areas
  • Added practical Python and C++ examples throughout

Built-in modules and protocol coverage

  • Consolidated hardware, protocols, and utilities under built_in_modules
  • Expanded documentation for DMA wrappers, raw memory mapping, OS memory bridge, SQL, version helpers, GPIB, PRBS, file I/O, compression, HLS, and simulation helpers such as Pgp2bSim and MemEmulate
  • Expanded and cleaned up protocol sections including UDP, RSSI, SRP, Packetizer, Batcher, EPICS V4, Xilinx/XVC, UART, network wrappers, and GPIB
  • These pages now include clearer conceptual overviews, stronger integration guidance, code-backed examples, lifecycle notes, and better cross-links

API reference improvements

  • Added conceptual links from API pages back to canonical narrative docs
  • Regrouped Python API docs around actual package structure and user-facing topics
  • Cleaned up many C++ and Python API wrapper pages to better reflect the new docs structure
  • Standardized backlink language around Related Topics
  • Expanded missing Python and C++ API coverage for a number of Rogue classes and bindings
  • Added generated Boost.Python API documentation support for binding-heavy areas

Generated API tooling and source links
This branch now includes substantial docs tooling work in addition to content rewrites:

  • Added custom Sphinx extensions for Boost.Python API generation/rendering
  • Fixed the custom Boost.Python directive for Sphinx 8
  • Added local Doxygen-backed source links for C++ API docs
  • Enabled Doxygen source browsing and .cpp indexing so C++ API entries can link to declarations and implementations
  • Added a docs-only Doxygen input filter to normalize selected .cpp alias spellings during docs generation
  • Added a first-pass Binding source link for Boost.Python-generated API pages so wrapper .cpp sources can be previewed in built docs

In practice this means:

  • C++ API entries can show [header] and, where available, [impl]
  • Boost.Python-generated pages can show a Binding source link
  • These links target local generated Doxygen HTML, not GitHub

Navigation behavior

  • API sections now expand deeply in the sidebar so nested namespaces/packages are actually navigable
  • Non-API sections stay shallower so the general docs do not become fully exploded
  • API page-body toctrees remain shallow for readability
  • Related Topics remains visible in page content without polluting API sidebar navigation

Tutorials, cookbook, and logging

  • Expanded tutorials, cookbook content, and supplemental getting-started guidance
  • Added more practical examples for advanced stream patterns, PyRogue devices/variables, system integration, and device workflow
  • Reworked logging documentation substantially and added logger guidance across many module pages

Content migration
This branch performs the bulk of the documentation migration away from older locations such as:

  • docs/src/interfaces/*
  • older protocols/*, hardware/*, and utilities/* layouts
  • fragmented older PyRogue structural layouts

Legacy material was moved, merged, rewritten, or deleted depending on whether it still served a useful purpose.

Suggested review focus
Because of the size, the most effective review path is:

  1. Top-level navigation and information architecture
  2. pyrogue_tree, stream_interface, and memory_interface
  3. built_in_modules
  4. protocol sections and tutorial/cookbook additions
  5. API reference structure, generated Boost.Python pages, and source-link behavior

…avigation. Updated index files to reflect new paths, added conceptual overviews for memory and stream interfaces, and introduced new sections for tutorials and advanced patterns. Enhanced existing documentation with links to related content and clarified transaction lifecycle details. Added new pages for cookbook recipes and built-in modules.
…y. Changed "Class Documentation" to "API Reference" across multiple files, enhancing the terminology used in the documentation. Added conceptual guidance links for memory interfaces and updated various sections for better navigation.
… for model types and updating references across related files. Improved navigation with links to API reference pages and conceptual guidance, ensuring clarity for users interacting with memory models in PyRogue.
…ing conceptual guidance sections with links to relevant resources. This update improves clarity and navigation for users, covering topics such as memory models, stream architecture, command behavior, and variable usage.
…oducing a new 'Built-in Modules' index. Updated references in existing documentation to enhance navigation and clarity, ensuring users can easily access relevant resources.
…ration phase. Updated the migration matrix with new content paths, added narrative preservation rules, and established canonical pages for client and memory interfaces. Expanded sections on protocols, quick start, and tutorials to provide clearer guidance and structure for users. Ensured legacy content remains accessible while improving overall documentation coherence.
Introduced a new page for built-in integrations, consolidating various integration-oriented features. Updated the main index to include a reference to this new section. Adjusted existing documentation for simulation, SQL logging, and version helper interfaces to indicate their migration to the new integrations page, ensuring clarity during the documentation revamp process.
Introduced a new overview page for built-in modules, consolidating information on utilities, hardware, and protocols. Updated the main index to include a reference to this new overview section. Adjusted migration matrix to reflect the movement of existing narratives to the new overview page, ensuring clarity during the documentation revamp process.
Added new sections for logging and PyDM, and updated the main index to include these entries. Enhanced the migration matrix with additional narrative paths for installing, logging, and PyDM documentation. Retained legacy content with clear indications of their migration to canonical operational pages, ensuring users have access to relevant resources during the documentation revamp process.
…ious protocols including UDP, Packetizer, and EPICSV4. Each protocol page now includes a status section indicating legacy content retention and plans for future narrative expansions. This update ensures users are informed about ongoing documentation changes during the migration process.
… protocol page now includes a note indicating retention during migration and provides links to conceptual guidance. This ensures users are informed about the ongoing documentation changes and can easily access relevant resources.
…terfaces. Added links to relevant resources for memory, stream, and protocol usage, enhancing clarity and navigation for users during the migration process.
…g. Added numerous new paths and narrative sections for various interfaces, including built-in modules and protocols, ensuring users have clear guidance during the documentation revamp process. Enhanced clarity by indicating legacy content retention and migration actions for improved navigation.
…nt retention and plans for future expansions. This update enhances clarity for users navigating the ongoing documentation revamp process.
…e status. The new M2 Closeout section summarizes completed objectives and outlines deferred tasks for M3, while the policies document now includes a record of the M2 freeze, emphasizing legacy content retention and navigation structure. This enhances clarity for users during the ongoing documentation revamp process.
Introduced 'Common Quick Checks' and 'First Success Path' to the Quick Start guide, providing users with essential setup validation and a streamlined path to getting started with PyRogue. Added 'Device Workflow Tutorial' and 'System Integration Tutorial' to the Tutorials section, offering comprehensive guidance on building custom devices and integrating systems. Updated the respective index files to include these new sections, enhancing navigation and user experience during the documentation revamp process.
…ge scenarios for UDP, EPICSV4, and Packetizer protocols. Added configuration checklists and related documentation links to improve user guidance during the ongoing documentation revamp process. Updated migration matrix to reflect these additions and narrative expansions.
…atterns and PyRogue device management. Added detailed procedures for stabilizing bursty producers and creating computed status values, along with design principles and common failure modes for PyRogue usage. Updated index files to improve navigation and user experience during the documentation revamp process.
…4, and SRP protocols. Added detailed usage notes, version compatibility guidelines, and integration patterns to improve user understanding. Updated migration matrix to reflect these additions and narrative expansions, ensuring clarity during the ongoing documentation revamp process.
… and lifecycle management. Added practical notes on command context and clarified the Managed Interface Lifecycle, improving user understanding of interface interactions. Updated related integration references for better navigation.
…ecycle management for RSSI, SRP, and XVC protocols. Added details on thread behavior, state management, and locking mechanisms to improve user understanding of protocol operations. Updated integration references for better navigation and clarity during the documentation revamp process.
…reference structure. Introduced a clear information architecture, defined content placement rules, and outlined implementation milestones to enhance accessibility and coherence for new and experienced users. This plan aims to streamline the documentation journey while maintaining reference quality and stability during the migration process.
…ject. This new file outlines the canonical plan, milestone status, key decisions, protocol-specific decisions, and a suggested execution order for remaining tasks. It serves as a guide for future contributors to ensure continuity and clarity in the documentation process.
…tion models, threading, and lifecycle management. Added code-backed examples for practical usage and clarified integration notes to improve user understanding and navigation during the documentation revamp process.
…e clarity. Updated sections for Client, Server, and Core classes with detailed descriptions, usage patterns, and code-backed examples. Improved navigation by linking related documentation and providing practical checklists for deployment scenarios, ensuring better user guidance during the documentation revamp process.
…lity. Updated core and coreV2 sections with detailed behavior summaries, usage patterns, and code-backed examples. Improved navigation by linking related documentation and clarifying when to use each core version, ensuring better guidance for users during the documentation revamp process.
…d lifecycle management. Added details on the internal controller's role in managing protocol-state execution and link management for both Client and Server. Improved clarity on the Managed Interface Lifecycle and concurrency-sensitive operations to aid user understanding during the documentation revamp process.
… enhance clarity. Updated sections for Application, Controller, ControllerV1, ControllerV2, Core, CoreV2, and Transport with concise descriptions of their roles and reference guidance. Improved navigation by linking to conceptual context and version-selection notes, ensuring better user understanding during the documentation revamp process.
…placeholder C++ API wrappers to reference-oriented pages and expanding protocol sections with code-backed content. Update migration matrix to reflect cleanup progress and regroup Python API documentation by core roles. Improve narrative-to-reference cross-linking and outline next steps for M4 preparation.
Update MethodDoc and HeaderDoc classes to enhance documentation parsing by introducing brief, details, params, and returns fields. Modify the parse_doxygen_comment function to return a ParsedComment object, improving the handling of Doxygen comments. Adjust related functions to accommodate these changes, ensuring better organization and clarity in API documentation.
Enhance the normalization functions for Doxygen comments by improving the handling of paragraph and bullet point formatting. The normalize_doc_block function now better processes text blocks, ensuring cleaner output for brief and detailed sections in API documentation. Additionally, adjustments in rendering methods ensure consistent formatting for bullet points, improving overall clarity in generated documentation.
…n_core.py

Add support for constructor argument documentation in the ClassDoc structure, improving the clarity of API references. Update the render_class_section function to conditionally include constructor arguments in class signatures. Additionally, refactor method rendering to support indentation, enhancing the overall formatting of method entries in the documentation. Update RST files to utilize the new rogue_boostpython_api directive for various classes, ensuring consistent documentation across the API.
…ion and rendering

Add new fields for constructor parameters and C++ target/role in ClassDoc, enhancing the clarity of API documentation. Update the parse_binding function to handle constructor documentation more effectively, including support for method entries and improved indentation in rendered output. Refactor related rendering functions to ensure consistent formatting across the documentation. Additionally, standardize the implementation section headers in various RST files for better organization.
Refactor various RST files to enhance the organization of Python API documentation. This includes updating paths to follow the full namespace structure, increasing the maxdepth of toctrees for better navigation, and adding new sections for EPICS V4 protocol wrappers and file I/O utilities. Additionally, standardize references to various classes and improve the overall consistency of documentation across multiple files.
…spaces to expand fully. Set 'navigation_depth' to -1 for improved accessibility of nested content.
… of M4 harmonization and release-freeze state. Add post-merge follow-up list for deferred tasks, enhance clarity in various sections, and finalize milestone artifacts for merge readiness.
Refactor multiple RST files to improve the organization of API documentation. This includes adding "Related Topics" sections to various files, ensuring consistent references across Python and C++ documentation. Additionally, update conceptual usage guidance for better user navigation and understanding of related content.
…rovements

Introduce a new `WINDOW_HANDOFF_DOCS.md` file detailing the purpose, user instructions, and branch state for the documentation handoff process. Enhance sidebar navigation by implementing a custom JavaScript file to manage API sidebar behavior, ensuring deep expansion for API sections while maintaining a shallower navigation for non-API pages. Update `conf.py` to include the new JavaScript file and adjust multiple RST files to standardize "Related Topics" sections and improve the overall structure of API documentation.
Update multiple RST files across Python and C++ API documentation to enhance consistency and clarity. This includes restructuring "Related Topics" sections for improved navigation and ensuring uniformity in the presentation of related content. Adjustments were made to the toctree captions and overall organization to facilitate better user experience and accessibility of information.
Update the parse_binding function in rogue_boostpython_core.py to improve the handling of constructor documentation. The logic now prioritizes class-specific method documentation over the generic "create" method, enhancing clarity and consistency in API documentation. This change ensures that constructor parameters and descriptions are accurately captured and rendered.
…ignatures and enabling source browsing in Doxygen. Update Doxyfile to include source files and modify Sphinx configuration to integrate new source link functionality. Additionally, apply CSS styles for improved presentation of source links in the documentation.
…lings. Update Doxyfile to apply the filter for .cpp files and add a new Python script for alias expansion, enhancing documentation accuracy.
…cumentation. Increment last updated date, add local source-code links for C++ API docs, enable Doxygen source browsing, and implement a custom input filter for alias normalization. Fix a typo in Doxygen parameter names and document expected C++ source link behavior.
…rt for source links to C++ bindings. Implement functions to normalize repository paths and load Doxygen file mappings, enabling accurate linking to source files in the generated documentation. Update the rendering logic to include binding source URIs where applicable.
…mplement a mechanism to preview wrapper source files in generated API pages, including a new `Binding source` line. Fix compatibility issue with Sphinx 8 directive handling. Update documentation to reflect these changes and clarify the provisional nature of the new feature.
…the documentation revamp. This includes the deletion of `DOCS_REVAMP_HANDOFF.md`, `PLAN.md`, `WINDOW_HANDOFF_DOCS.md`, and various related files in the `documentation_plan` directory, which have been superseded by finalized artifacts and are no longer needed. The branch is now in a release-freeze state, ready for merge into `pre-release`.
@bengineerd bengineerd marked this pull request as ready for review March 12, 2026 02:26
@bengineerd bengineerd requested review from ruck314 and slacrherbst and removed request for slacrherbst March 12, 2026 02:26
@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 29.59%. Comparing base (455967d) to head (6b7ed3d).
⚠️ Report is 788 commits behind head on pre-release.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@               Coverage Diff               @@
##           pre-release    #1131      +/-   ##
===============================================
+ Coverage        29.54%   29.59%   +0.04%     
===============================================
  Files               65       68       +3     
  Lines             6644     6944     +300     
  Branches          1015      969      -46     
===============================================
+ Hits              1963     2055      +92     
- Misses            4527     4736     +209     
+ Partials           154      153       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@bengineerd bengineerd merged commit f2cf8e7 into pre-release Mar 16, 2026
6 checks passed
@ruck314 ruck314 deleted the doc-update-6 branch April 14, 2026 17:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants