UX Overhaul: CLI, templates, TUI, local mode, and docs - #226

Implement the E2E test harness and validation framework as the foundation
for validating all future UX improvements. This infrastructure prevents
regressions and serves as living documentation.

E2E Test Harness (madsci.common.testing):
- types.py: Pydantic models for test definitions, steps, and validations
- validators.py: 14 validator types (exit_code, file_exists, http_health,
  json_contains, python_syntax, ruff_check, etc.)
- runner.py: E2ETestRunner with pure Python and Docker mode support
- template_validator.py: Template validation with syntax/lint checking

Test Infrastructure:
- 43 passing unit tests for validators and runner
- tests/e2e/ directory for tutorial test definitions
- YAML-based test definitions for human-readable test cases

CI Integration:
- .github/workflows/validation.yml for automated validation
- Runs E2E framework tests, tutorial tests, and template validation

Code Quality:
- All code passes ruff linting with no warnings
- Full docstrings on public classes and methods
- Refactored complex methods for maintainability

Documentation:
- docs/guides/ux_overhaul_progress.md tracking implementation status

This update establishes a clear separation between nodes (runtime servers
that communicate with workcells) and modules (complete packages containing
node, interfaces, drivers, types, tests, and deployment configs).

Key changes:
- CLI: Add `madsci new module` and `madsci new interface` commands
- Templates: Restructure from node/ to module/ hierarchy with interface variants
- TUI: Update wizard from "New Node" to "New Module" with interface selection
- Registry: Add "module" as a component type with interface_variants metadata
- Migration: Add `madsci migrate module` for standardizing external repos
- Settings: Add ModuleSettings, InterfaceSettings, and foo_types.py pattern

This clarifies the Equipment Integrator workflow and enables testing
without hardware via fake/sim interfaces.

- Add concrete deprecation timeline: deprecated in v0.7.0, removed in v0.8.0
- Define TUI phased delivery: MVP in Phase 1, full features in later phases
- Add multi-workcell support to ID Registry design
- Add air-gapped environment support for template installation
- Improve docker-compose migration safety (ruamel.yaml, validation, rollback)
- Add OTEL integration notes across CLI, TUI, and ID Registry designs
- Correct documentation URL to https://ad-sdl.github.io/MADSci/
- Add madsci run workflow convenience command
- Clarify Windows compatibility requirements (filelock, pathlib)
- Add schema export capability to Settings Consolidation
- Add per-step timeout support to E2E test harness
- Expand ExperimentCampaign design notes

Add unified madsci CLI entry point with the following commands:
- madsci version: Display installed MADSci packages and versions
- madsci doctor: System diagnostics (Python, Docker, ports)
- madsci status: Service health status with watch mode
- madsci logs: Log viewing from Event Manager with filtering
- madsci tui: Launch interactive Textual-based TUI

Core infrastructure includes:
- AliasedGroup for command aliases support
- MadsciCLIConfig for TOML-based configuration
- Rich-based output formatting utilities
- Textual TUI with dashboard, status, and logs screens

Also adds comprehensive test suite for CLI commands.

This commit completes Phase 2 of the MADSci UX overhaul plan:

## ID Registry System
- Add LocalRegistryManager for file-based name-to-ID mappings
- Add LockManager with heartbeat-based distributed locking
- Add IdentityResolver for lab-level coordination
- Add registry CLI commands (resolve, list, export, import, clean)

## Migration Tools
- Add MigrationScanner to detect definition files needing migration
- Add MigrationConverter to convert definitions to new format
- Add MigrationRollback for safe rollback capability
- Add migrate CLI commands (scan, convert, status, finalize, rollback)

## Template System
- Add TemplateEngine for rendering Jinja2 templates
- Add TemplateRegistry for discovering bundled/user templates
- Add template types with validation and hooks

## Settings Consolidation
- Add ModuleSettings and NodeModuleSettings for node development
- Add interface settings hierarchy (Serial, Socket, USB, HTTP)
- Add settings export endpoint to AbstractManagerBase

## Deprecation Layer
- Add deprecation utilities with timeline (deprecated v0.7.0, removed v0.8.0)
- Integrate warnings into manager definition loading

## Code Quality
- Fix all ruff linting issues (36 errors)
- Convert all f-string logging to structured kwargs (37 violations)
- Add comprehensive test coverage for new components

Add madsci new command group with subcommands for creating MADSci
components from templates:
- madsci new module: Create complete module with node, interfaces, types
- madsci new interface: Add interface variants to existing modules
- madsci new node/experiment/workflow/workcell/lab: Component scaffolding
- madsci new list: Discover available templates

Add bundled templates in madsci_common:
- module/basic: Complete module with fake interface for testing
- interface/fake: Simulated interface for testing without hardware
- experiment/script: Simple run-once experiment
- workflow/basic: Single-step workflow YAML
- workcell/basic: Workcell configuration
- lab/minimal: Lab configuration without Docker

All commands support interactive and non-interactive modes with
Rich prompts for parameter collection and file preview.

Includes 11 CLI tests, all passing.

Refactor experiment execution infrastructure to support different execution
contexts using composition over inheritance:

- ExperimentBase: Core class using MadsciClientMixin composition with
  lifecycle methods (start, end, pause, cancel, fail) and manage_experiment()
  context manager

- ExperimentScript: Simple run-once experiments with run() and main() methods

- ExperimentNotebook: Jupyter notebook support with start()/end() pattern,
  run_workflow() convenience method, and Rich display integration

- ExperimentTUI: Interactive terminal UI using Textual (optional dependency)
  with status display, log viewer, and control buttons

- ExperimentNode: Server mode exposing run_experiment as REST API action

- Updated experiment templates to use new modalities
- Added deprecation warning to ExperimentApplication (deprecated v0.7.0,
  removed v0.8.0)
- 31 tests for all new modalities

Reduce test suite runtime from ~32 minutes to ~6 minutes (5.2x faster)
by changing cleanup fixtures from function to module scope.

Key changes:
- Root conftest.py: Add module-scoped cleanup_resources_after_module fixture
- Root conftest.py: Only run gc.collect() between modules, not every test
- madsci_node_module conftest.py: Change cleanup_temp_files to module scope
- madsci_client conftest.py: Change cleanup_temp_files and
  cleanup_logging_handlers to module scope
- test_stress.py: Mark test_burst_traffic with @pytest.mark.slow
- test_middleware.py: Reduce rate limit windows for faster tests

The function-scoped fixtures were causing massive I/O overhead
(989s system time) from running glob operations, file deletion, and
gc.collect() after every single test (~1920 tests). Module scope
reduces this to ~50-100 cleanup operations instead.

Also includes EventClient test isolation fixes to prevent tests
from attempting to connect to real event servers.

Add comprehensive tutorials and persona-based guides:

- Tutorials (docs/tutorials/):
  - 01-exploration.md: CLI, TUI, and concepts introduction
  - 02-first-node.md: Module creation with fake interface
  - 03-first-experiment.md: Experiment scripts and notebooks
  - 04-first-workcell.md: Multi-node coordination with workflows
  - 05-full-lab.md: Complete lab deployment with Docker

- Equipment Integrator Guide (docs/guides/integrator/):
  - README.md: Guide overview and quick reference
  - 01-understanding-modules.md: Node/Module/Interface concepts
  - 02-creating-a-module.md: Module scaffolding walkthrough
  - 03-developing-interfaces.md: Interface patterns
  - 04-fake-interfaces.md: Simulated interface patterns

- Lab Operator Guide (docs/guides/operator/):
  - README.md: Quick reference for daily operations

- Experimentalist Guide (docs/guides/experimentalist/):
  - README.md: Quick reference for running experiments

- Tutorial automation:
  - tutorial_02_first_node.tutorial.yaml: E2E test for module creation

- Updated myst.yml with new navigation structure
- Updated ux_overhaul_progress.md to reflect Phase 5 progress (75%)

Fix 3 failing tests in tests/e2e/test_tutorials.py by correcting the
tutorial YAML files to match the E2ETestDefinition Pydantic schema and
fixing a bug in the template engine where rendered source paths were
passed to Jinja2 get_template() instead of the original unrendered
paths (which match the actual filenames on disk with {{variable}}
placeholders).

…tes, and template tests

CLI improvements:
- Fix migrate.py and registry.py to respect global --no-color/--quiet/--json flags
- Replace module-level Console() with context-aware _get_console(ctx)
- Add @click.pass_context to all migrate (5) and registry (6) subcommands
- Replace raise SystemExit(1) with ctx.exit(1) for proper Click lifecycle
- Replace literal Unicode characters with escape sequences
- Fix variable shadowing (error -> err) in migrate commands

New templates (8 -> 11 total):
- experiment/tui: ExperimentTUI modality with pause/cancel support
- experiment/node: ExperimentNode modality with REST server
- workflow/multi_step: 3-step workflow with node parameters
- module/device: 11-file device module using @action decorator pattern,
  resource management with Slots, full device lifecycle, fake interface
  with command_history tracking

Bug fix:
- Fix workflow/basic template referencing undefined author_name variable

Tests:
- Add 74 template engine/registry tests across 5 test classes
- Validate all 11 templates for existence, defaults, rendering, and syntax

… CLI imports, and E2E tests

Add missing node/basic template that was referenced by CLI but empty,
enhance lab/minimal template with .gitignore, pyproject.toml, and example
workflow. Implement lazy imports across all CLI commands for faster startup
performance. Add 3 new E2E tutorial tests covering workflow/workcell/lab
creation and full lifecycle validation. Update template tests accordingly.

…bility, and documentation accuracy

Fix 3 failing E2E tutorial tests:
- Fix generate_from_template() to map --name flag to all template
  parameter types (lab_name, workflow_name, workcell_name) using
  category-based resolution instead of hardcoded parameter list
- Fix device module template E501 lint error (line too long)
- Fix tutorial_05 Python 3.10 compatibility (tomllib fallback)

Fix documentation accuracy across tutorials and guides:
- Replace non-existent --output flag with positional DIRECTORY arg
- Fix ExperimentDesign field names and import paths
- Fix run() -> run_experiment() method override
- Fix --template -> --modality for experiment CLI
- Fix invalid --check all, --service, --verbose CLI flags
- Fix broken workflow_schema.md link

Add .scratch/ to .gitignore for local testing.

…, redaction, utcnow, deprecation warnings, lazy imports

Resolves all issues identified in the ux_overhaul merge code review:

- CR-1 (High): Fix stdlib logging with kwargs in 7 files (~37 call sites).
  Converted structlog-style logger.warning("msg", key=val) to stdlib-compatible
  logger.warning("msg: key=%s", val) pattern to prevent runtime TypeError.

- CR-2 (Medium): Fix _sync_to_lab in identity_resolver.py to include
  component_id in the POST body, enabling distributed ID coordination.

- CR-3 (Medium): Fix over-aggressive settings redaction in manager_base.py.
  Replace broad substring matching with word-boundary-aware patterns to
  avoid redacting legitimate fields like primary_key or auth_enabled.

- CR-4 (Medium): Replace all datetime.utcnow() calls across 6 files with
  datetime.now(tz=timezone.utc) for Python 3.12+ compatibility. Uses
  timezone.utc (not UTC constant) for Python 3.10 compatibility.

- CR-5 (Low): Fix DeprecatedClass double-warning on nested subclasses by
  checking cls.__dict__ instead of hasattr() and tracking wrapped state.

- CR-6 (Low): Implement true lazy CLI command loading via AliasedGroup.
  get_command() with _LAZY_COMMANDS registry, replacing eager imports
  that defeated the stated lazy loading intent.

All 2004 tests pass.

The structlog ConsoleRenderer with colors=True was writing ANSI escape
codes to both console and file handlers. Add AnsiStrippingFormatter to
the file handler so log files remain clean while console output stays
colorized.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Fix double-escaped unicode in doctor command that printed literal "\u2713"
instead of the checkmark character. Add version field to ManagerHealth and
populate it via the health endpoint using importlib.metadata, with a
fallback for __main__ module resolution so it works when managers are run
with python -m.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add 59 new tests covering the three CLI commands that previously had zero
dedicated tests (registry, migrate, tui), plus integration tests validating
the migration tool against example_lab. All 2063 tests pass.

- test_registry.py: 21 tests covering all 6 subcommands (list, resolve,
  rename, clean, export, import) with empty/populated registries, JSON
  output, type filtering, and error cases
- test_migrate.py: 21 tests covering all 5 subcommands (scan, convert,
  status, finalize, rollback) with empty dirs, definition files, and
  edge cases
- test_migrate_example_lab.py: 12 integration tests validating the scanner
  finds all 20 definition files (7 manager + 6 node + 7 workflow), extracts
  component IDs, plans migration actions, and dry-run doesn't modify files
- test_tui.py: 5 tests covering help, screen choices, mocked launch,
  graceful import error handling, and the 'ui' alias
- Update completion plan to mark Phase A complete, unblocking Phase B
- Add .scratch directory directive to AGENTS.md

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Adds start, stop, init, validate, run, completion, and backup commands
to the madsci CLI, bringing CLI command coverage from 53% to 100%.
All commands follow existing patterns (lazy loading, Click, Rich output)
and include dedicated test files (47 new tests, 2110 total passing).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…est coverage

Add 14 new templates across 4 categories (module, interface, lab, comm),
bringing the template library from 12 to 26 templates. Add COMM category
to TemplateCategory enum. Expand template test suite from ~85 to 150 tests
covering existence, defaults validation, rendering, and Python syntax for
all templates.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…eming, and command palette

- D.1: Trogon command palette via `madsci commands` CLI + Ctrl+P in TUI
- D.2: Auto-refresh (5s) on Dashboard and Status screens with 'a' toggle
- D.3: Node management screen (nodes from Workcell Manager, detail panel)
- D.4: Workflow visualization screen (active/queued workflows, step progress, pause/resume/cancel)
- D.5: External CSS theming (styles/theme.tcss), removed all inline DEFAULT_CSS
- Updated app.py with 5 screens, new keybindings (n/w/Ctrl+P), updated help
- 14 TUI tests (up from 5), 2119 total tests passing, ruff clean

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add migration guide (docs/guides/migration.md) documenting the transition from
definition YAML files to settings+registry configuration. Create 8 integration
tests that run the converter against a copy of example_lab, validating backup
creation, ID registration, deprecation markers, and rollback. Update
example_lab/.env with commented-out settings-equivalent environment variables.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…rides

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Enable running the full MADSci stack without Docker via `madsci start --mode=local`.
All 7 managers run in a single process with in-memory replacements for Redis,
MongoDB, and PostgreSQL (SQLite), requiring zero changes to manager business logic.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…ification

Replace implicit auto-writing of definition/info files with explicit CLI
commands and metadata-driven secret handling. G.1 adds model_dump_safe()
and json_schema_extra={"secret": True} annotations across all manager
settings. G.2-G.3 remove auto-writing behavior (managers/nodes). G.4 adds
madsci config export/create commands. G.5 adds NodeInfo.from_config()
factory with identity fields on NodeConfig. G.6 adds migration guide and
deprecation warnings. 54 new tests, 2359 total passing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Workflows are data that flows between components, not long-lived
processes managing their own identity. The migration tooling for
workflows was essentially a no-op, creating misleading output that
marked workflow files as "deprecated" without migrating them anywhere
useful. This simplifies the migration tool to only handle managers
and nodes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

MadsciContext (a MadsciBaseSettings subclass) was instantiated eagerly in
the Click group callback, causing Pydantic's CliSettingsSource to parse
sys.argv and intercept --help before Click could handle it. Replace eager
instantiation with a lazy proxy that defers MadsciContext creation until a
command callback actually accesses it.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Extract structural data from manager definition files into standalone
YAML files and inline settings, removing all *_manager_definition
references from settings.yaml:

- locations.yaml: 10 location definitions from location manager
- transfer_capabilities.yaml: transfer templates and routing config
- resource_templates.yaml: plate_nest and storage_stack templates
- workcell_nodes inline in settings.yaml: 5 node URLs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Move example lab and notebooks under examples/ for cleaner top-level
structure. Relocate developer guides (node development, workflows,
observability, troubleshooting) from example_lab/ into docs/guides/
where they belong. Update all path references in compose files,
justfile, myst.yml, AGENTS.md, READMEs, tutorials, and CLI help text.

Also adds v0.6 migration test fixtures and removes stale files
(test_admin_commands.py, test.txt, duplicate notebook).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Remove DefinitionT type parameter from AbstractManagerBase, making all
managers use settings-based configuration exclusively. Manager identity
(manager_id) is now Optional in settings with runtime ULID generation,
and registry resolution provides stable IDs. Definition types remain
with deprecation warnings for migration tooling only.

Key changes:
- AbstractManagerBase[SettingsT] (single type param, no more DefinitionT)
- Remove load_definition(), DEFINITION_CLASS, definition file I/O
- All 7 manager servers updated to use self.settings.* instead of self.definition.*
- AbstractNode uses NodeConfig directly (no NodeDefinition file loading)
- Delete example lab definition files (managers/*.yaml, node_definitions/*.yaml)
- Add deprecation warnings on ManagerDefinition/NodeDefinition types
- Update CLI validate command with deprecation messaging
- Fix all tests (2460 passing, 0 failures)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Remove 3 remaining calls to the deleted _sync_locations_to_definition()
method in LocationManager (delete_location, set_representations,
remove_representation). Remove 6 xfail markers from test_location_server.py
that were guarding against these dead calls. Rewrite
test_location_persistence.py to test Redis-based persistence and settings
export instead of the removed YAML file round-tripping.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- Replace self.node_definition with self.node_info in example modules
  (liquidhandler, platereader, robotarm) to match purged NodeDefinition
- Update node_notebook.ipynb from NodeDefinition to RestNodeConfig pattern
- Update experiment_notebook.ipynb from ExperimentApplication to
  ExperimentScript, fix protocol paths for notebook CWD, add conditional
  pip install
- Fix backup_and_migration.ipynb backup paths to use ~/.madsci/backups/
  instead of relative paths that fail from notebook CWD
- Fix workcell_engine.py initialization order (init state before
  get_workcell_definition)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- Create examples/example_experiment.py using ExperimentScript with a
  complete workflow loop (liquid handler → plate reader) and automatic
  experiment lifecycle management
- Update experiment_notebook.ipynb to use ExperimentNotebook with the
  cell-by-cell start()/end() pattern, display() for rich output, and
  cross-references to the script example
- Add e2e tests validating syntax, imports, linting, and structural
  correctness of both the script and notebook

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- Create examples/example_experiment_tui.py demonstrating the TUI
  modality with interactive pause/cancel support via
  check_experiment_status()
- Add TestExampleExperimentTUI test class validating syntax, imports,
  structure, and TUI-specific patterns (run_tui, check_experiment_status)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- Make on_button_pressed async so it properly awaits the async action
  methods (start/pause/cancel) - fixes buttons not working
- Add periodic status refresh via set_interval while experiment is
  running so the experiment ID and status update in real time

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

action_start was awaiting the experiment coroutine directly, which
blocked the entire event loop and prevented pause/cancel/quit from
being processed. Refactored to use asyncio.create_task so the
experiment runs as a background task and the TUI stays responsive.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Use threading.Event objects for direct in-process communication between
the TUI and experiment thread instead of relying on server round-trips.
Cancel now immediately stops waiting via asyncio task cancellation.
Pause button toggles to "Resume" label when experiment is paused.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Enable MadsciBaseSettings subclasses to resolve config files relative to
a "settings directory" instead of CWD, with each filename walking up the
directory tree independently. This allows shared configs (settings.yaml
in a lab root) to coexist with per-instance configs (node.settings.yaml
in a node subdirectory).

Activation is opt-in only via _settings_dir kwarg, MADSCI_SETTINGS_DIR
env var, or --settings-dir CLI option. Without either, existing
CWD-relative behavior is preserved exactly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The experiment e2e tests were failing because ExperimentBase._setup_lab_context()
only set a ContextVar with discovered URLs but didn't store them on the instance.
When the Jupyter kernel ran from a directory without settings.yaml, the client
factory methods couldn't find server URLs. Now _setup_lab_context() also propagates
URLs to instance attributes, making client initialization more robust.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…by madsci_dashboard

- Replace eval() with simpleeval for skip_if expressions (critical security fix)
- Fix .env escaping for newline/carriage-return/tab characters
- Fix migration rollback to actually delete generated files and verify restores
- Add thread safety to ValidatorRegistry singleton with double-checked locking
- Fix type: ignore on ExperimentBase.config with Optional annotation
- Replace silent contextlib.suppress(Exception) with logged warning
- Add configurable pause timeout (max_pause_wait) with exponential backoff
- Rename test_pr_review_fixes.py to test_security_regressions.py
- Fix test isolation in test_experiment_modalities.py using nonlocal
- Add output path validation warnings for config export commands

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…Definition with WorkcellInfo

Definition files are now hard-deprecated in v0.7.0 (previously soft-deprecated
with removal planned for v0.8.0). WorkcellManagerDefinition is replaced by
a lightweight WorkcellInfo model for runtime state, while all configuration
comes from WorkcellManagerSettings. Also removes the deprecated example_app.py
and the unused lab_definition_path parameter from LocalRunner.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>