Feature: Complete Event Manager Improvements - #219
Merged
Conversation
Bumps ghcr.io/devcontainers-extra/features/ruff from 1.1.1 to 2.0.0. --- updated-dependencies: - dependency-name: ghcr.io/devcontainers-extra/features/ruff dependency-version: 2.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
…devcontainers-extra/features/ruff-2.0.0 Bump ghcr.io/devcontainers-extra/features/ruff from 1.1.1 to 2.0.0
This was
linked to
issues
Jan 30, 2026
Mark Phase 1 complete and note Phase 2 progress/remaining work.
Expose OTEL status in /health output and add OTEL settings to ManagerSettings to drive manager-level tracing configuration.
Add domain spans for workflow execution, per-step execution, and node action dispatch to improve end-to-end trace visibility.
Document WorkcellManager domain spans (workflow.execute/workflow.step/node.action) and update acceptance criteria notes.
Wrap datapoint ingestion in a data.save span and update Phase 2 progress notes.
Add domain spans for resource lifecycle/query endpoints and for location creation, resource attachments, and transfer planning.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR delivers a broad observability overhaul across MADSci: introducing process-wide OpenTelemetry bootstrapping, a hierarchical EventClient context system, and a structured-logging migration backed by new tests and example-lab infrastructure.
Changes:
- Added canonical OTEL bootstrap/helpers (
madsci.common.otel) plus manager/client runtime integration, spans, and example-lab observability stack. - Introduced
contextvars-based EventClient context propagation (middleware + client mixin integration) for hierarchical structured logging. - Added logging/audit tooling and expanded EventType usage, with extensive new tests and docs.
Reviewed changes
Copilot reviewed 105 out of 227 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
| src/madsci_event_manager/madsci/event_manager/notifications.py | Migrates email alert logs to structured format with event_type. |
| src/madsci_event_manager/madsci/event_manager/events_csv_exporter.py | Adds CSV export logging (currently via stdlib logger). |
| src/madsci_data_manager/madsci/data_manager/data_server.py | Adds OTEL spans around datastore endpoints and structured startup logs. |
| src/madsci_common/tests/test_manager_context.py | Adds tests for manager + middleware context integration. |
| src/madsci_common/tests/test_manager_base_otel_runtime_reuse.py | Verifies OTEL runtime reuse behavior in manager base. |
| src/madsci_common/tests/test_event_types_phase3.py | Ensures EventType descriptions cover all enum members. |
| src/madsci_common/tests/test_event_type_coverage.py | Adds EventType coverage/pattern tests + validates logging hook/docs exist. |
| src/madsci_common/pyproject.toml | Adds OTEL API dependency and optional OTEL/instrumentation extras. |
| src/madsci_common/madsci/common/utils.py | Replaces f-string logging with structured/printf-style logging. |
| src/madsci_common/madsci/common/types/manager_types.py | Adds OTEL settings fields on ManagerSettings; adjusts ManagerType. |
| src/madsci_common/madsci/common/otel/requests_instrumentation.py | Adds best-effort requests auto-instrumentation helper. |
| src/madsci_common/madsci/common/otel/propagation.py | Adds helper to inject OTEL propagation headers. |
| src/madsci_common/madsci/common/otel/fastapi_instrumentation.py | Adds best-effort FastAPI auto-instrumentation helper. |
| src/madsci_common/madsci/common/otel/init.py | Exposes canonical OTEL bootstrap and helper APIs. |
| src/madsci_common/madsci/common/mongodb_version_checker.py | Migrates logs to structured pattern w/ exc_info=True. |
| src/madsci_common/madsci/common/middleware.py | Adds EventClientContextMiddleware + lazy context wrapper. |
| src/madsci_common/madsci/common/manager_base.py | Adds manager OTEL bootstrap, span helper, middleware wiring, structured logs. |
| src/madsci_common/madsci/common/context.py | Implements EventClient context APIs via contextvars. |
| src/madsci_common/madsci/common/backup_tools/postgres_backup.py | Migrates backup logs to structured EventType-based logging. |
| src/madsci_common/madsci/common/backup_tools/backup_validator.py | Migrates validator logs to structured EventType-based logging. |
| src/madsci_common/madsci/common/backup_tools/backup_manager.py | Migrates backup manager logs to structured EventType-based logging. |
| src/madsci_common/madsci/common/init.py | Installs rich traceback handler globally on import. |
| src/madsci_client/tests/test_event_client_otel_metrics_subprocess.py | Adds subprocess test for deterministic OTEL metrics assertions. |
| src/madsci_client/tests/test_event_client_otel.py | Adds EventClient OTEL integration tests (trace context + instruments). |
| src/madsci_client/tests/test_event_client_context.py | Adds comprehensive tests for EventClient context API + async propagation. |
| src/madsci_client/tests/test_client_context_integration.py | Adds tests ensuring clients/mixin participate in context inheritance. |
| src/madsci_client/pyproject.toml | Adds OTEL API + structlog dependency; OTLP exporter as optional extra. |
| src/madsci_client/madsci/client/workcell_client.py | Switches to context-aware logging client creation + structured error logs. |
| src/madsci_client/madsci/client/structlog_config.py | Adds per-instance structlog configuration with OTEL/hierarchy processors. |
| src/madsci_client/madsci/client/otel_processors.py | Adds structlog processors injecting OTEL trace context into logs. |
| src/madsci_client/madsci/client/node/rest_node_client.py | Adds context-aware logger creation + structured error logs. |
| src/madsci_client/madsci/client/location_client.py | Uses context-aware logger creation; adds warning event_type field. |
| src/madsci_client/madsci/client/data_client.py | Adds EventType to warnings in data client operations. |
| src/madsci_client/madsci/client/client_mixin.py | Makes mixin EventClient resolution context-aware + adds explicit teardown. |
| src/madsci_client/README.md | Documents context-based logging usage patterns for clients. |
| scripts/precommit_check_logging_patterns.py | Adds pre-commit hook to ratchet structured logging patterns. |
| ruff.toml | Adds excludes/ignores for scripts + markdown. |
| pyrightconfig.json | Adds pyright config with relaxed reporting settings. |
| pyproject.toml | Adds structlog/colorama dependencies; adds OTEL + pytest-asyncio to dev deps. |
| myst.yml | Updates docs TOC paths and adds docs/madsci_powered section. |
| madsci-entrypoint.sh | Ensures venv env vars are set during container entrypoint. |
| examples/otel_demo.py | Adds OTEL demo script using canonical bootstrap and trace correlation examples. |
| examples/README_OTEL_POC.md | Adds OTEL PoC README documentation. |
| example_lab/otel/prometheus.yaml | Adds Prometheus config for OTEL demo stack. |
| example_lab/otel/loki.yaml | Adds Loki config for OTEL demo stack. |
| example_lab/otel/grafana/provisioning/datasources/datasources.yaml | Provisions Grafana datasources for Jaeger/Prometheus/Loki. |
| example_lab/otel/grafana/provisioning/dashboards/dashboards.yaml | Provisions Grafana dashboard provider. |
| example_lab/otel-collector.yaml | Adds basic OTEL collector config (debug exporter). |
| example_lab/otel-collector-full.yaml | Adds full OTEL collector config (Jaeger/Prom/Loki exporters). |
| example_lab/notebooks/experiment_notebook.ipynb | Updates notebook code style + kernel metadata. |
| example_lab/example_transfer.py | Wraps example transfer script in EventClient context for hierarchical logging. |
| example_lab/example_app.info.yaml | Updates example app capabilities + rate limiting/uvicorn configuration schema. |
| example_lab/README.md | Adds link to real module examples and expands requirements wording. |
| example_lab/OBSERVABILITY.md | Adds end-to-end guide for the observability stack. |
| docs/madsci_powered/Modules.md | Adds list of integrated modules/devices. |
| docs/madsci_powered/Labs_and_Workcells.md | Adds list of MADSci-powered labs/workcells. |
| docs/madsci_powered/Experiments.md | Adds list of MADSci-enabled experiments. |
| docs/dev/opentelemetry_integration_plan/README.md | Adds OTEL integration plan index. |
| docs/dev/opentelemetry_integration_plan/14_phase_4_example_lab_infra.md | Documents example lab OTEL infra implementation details. |
| docs/dev/opentelemetry_integration_plan/10_phase_0_dependency_setup.md | Documents OTEL dependency strategy and acceptance criteria. |
| docs/dev/opentelemetry_integration_plan/02_current_state_assessment.md | Documents current OTEL/logging state assessment. |
| docs/dev/opentelemetry_integration_plan/01_overview.md | Documents OTEL plan overview and measurable goals. |
| docs/dev/opentelemetry_integration_plan/00_principles.md | Documents OTEL design principles (cardinality, precedence, shutdown). |
| docs/dev/logging_guidelines.md | Adds structured logging and context usage guidelines. |
| docs/dev/event_client_context_plan/README.md | Adds context system plan + quick reference. |
| docs/dev/event_client_context_plan/00_principles.md | Documents context system design decisions and naming conventions. |
| docs/api/madsci/experiment_application/index.md | Updates generated API docs output path/content. |
| docs/api/madsci/experiment_application/experiment_application.md | Updates generated API docs output path/content. |
| docs/api/madsci/common/types/resource_types/index.md | Updates generated API docs for resource types. |
| docs/api/madsci/client/resource_client.md | Updates generated API docs for resource client signatures. |
| docs/api/madsci/client/index.md | Updates generated API docs for client index signatures. |
| devbox.json | Adds devbox environment configuration (python/uv/pdm). |
| compose.yaml | Includes compose.otel.yaml and makes optional .env env_file. |
| compose.otel.yaml | Adds OTEL observability stack services (collector/jaeger/prom/loki/grafana). |
| README.md | Adds JOSS badge. |
| Dockerfile.dashboard | Adds venv verification steps during dashboard image build. |
| Dockerfile | Switches to PDM+uv venv install flow and installs OTEL extras. |
| .pre-commit-config.yaml | Adds local pre-commit hook for logging pattern enforcement. |
| .justfile | Writes generated pdoc output into docs/api/. |
| .github/workflows/gh_pages.yml | Updates pdoc output path to docs/api/. |
| .github/workflows/e2e_tests.yml | Increases compose startup wait to 60s for e2e. |
| .envrc | Adds direnv+devbox integration. |
| .env.example | Adds OTEL + retention/log-rotation configuration examples. |
| .devcontainer/devcontainer.json | Updates ruff devcontainer feature version. |
Comments suppressed due to low confidence (1)
src/madsci_common/madsci/common/types/manager_types.py:1
- This change removes the previous
ManagerType._missing_case-insensitive coercion behavior. If any YAML/env/config inputs relied on e.g.Event_Manager/EVENT_MANAGER/EventManager, they will now fail validation. If backward compatibility is a goal, reintroduce case-insensitive parsing (either via_missing_or a dedicated pydantic validator that normalizes input before enum parsing).
"""Types used primarily by MADSci Managers."""
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Add diagnostic steps to help debug CI failures: - Check service health endpoints before running tests - Show docker compose status and lab_manager logs pre-test - Show detailed service logs on test failure
Root cause: Docker Compose builds images in parallel, causing the dashboard image to pull the old published base image from GHCR instead of using the locally built one. This meant structlog and pytz were missing. Fixes: - Build base madsci image first before docker compose up - Add pytz to event_manager dependencies (was imported but not declared) - Update pdm.lock with pytz dependency
Changes: - e2e_tests now triggers after Docker Image workflow completes - Pulls pre-built images from GHCR instead of rebuilding - Adds docker workflow trigger for pull_request events - Properly handles PR tags (pr-<number>) and branch tags - Checkouts the correct commit for workflow_run events - Adds manual trigger option to build locally if needed Benefits: - Faster e2e tests (no image rebuild) - Ensures e2e tests use same images that will be published - Prevents build order issues between base and dashboard images
- Add concurrency group to docker workflow to cancel duplicate runs - When both push and pull_request events fire for the same ref, only one build will complete (the newer one cancels the older) - Update e2e_tests to reference renamed workflow 'Build Docker Images'
Use github.head_ref (for PRs) or github.ref_name (for push) so that both events for the same branch resolve to the same concurrency group. Example: PR from 'feature-x' and push to 'feature-x' both get group 'docker-feature-x', ensuring only one build runs.
Instead of using concurrency with cancel-in-progress (which causes cancelled workflows to show as failed checks), simply limit push triggers to main branch and tags. - PRs trigger via pull_request event (covers all feature branches) - Merges to main trigger via push event - Tags trigger via push event This eliminates duplicate runs without any cancelled/failed checks.
Documentation: - Create docs/guides/logging.md combining logging guidelines and context migration guide into a single production-ready document - Add Guides section to myst.yml table of contents - Add example_lab/OBSERVABILITY.md to TOC (already existed) - Remove docs/dev/ planning documents (19 files) Code cleanup: - Remove phase references from EventType comments - Rename test_event_types_phase3.py to test_event_type_descriptions.py - Update test docstring to remove phase reference All valuable documentation from the planning docs has been extracted into production docs. The OBSERVABILITY.md in example_lab already contained the complete user guide for the OTEL stack.
…ement This commit introduces class and function decorators to simplify context management across MADSci components: ## New Features ### Ownership Context Decorators (ownership.py) - @with_ownership: Function decorator for ownership context - @ownership_class: Class decorator that wraps all public methods - get_ownership_overrides() method support for dynamic ownership ### OpenTelemetry Tracing Decorators (otel/tracing.py) - span_context(): Context manager for creating spans - @with_span: Function decorator for tracing - @traced_class: Class decorator for automatic method tracing - Utility functions: get_tracer(), get_current_span(), set_span_error(), etc. ### MadsciContext Decorators (context.py) - @with_madsci_context: Function decorator for context management - @madsci_context_class: Class decorator for automatic context wrapping ## Refactoring ### LocationManager - Applied @ownership_class decorator - Removed 10 manual `with ownership_context():` blocks ### WorkcellEngine - Replaced manual tracer with span_context() from tracing module - Removed self._tracer initialization ### ResourceManager - Applied @ownership_class decorator with get_ownership_overrides() - Removed _setup_ownership() method - Removed HTTP middleware for ownership context ## Tests - 91 new decorator tests (ownership, context, tracing, stacking) - All existing tests pass (215 resource manager, 107 workcell, 101 location)
The ownership_class, madsci_context_class, with_event_client_class, and traced_class decorators were wrapping class attributes that are types (like DEFINITION_CLASS and SETTINGS_CLASS) as if they were instance methods. This caused TypeError when trying to instantiate these classes. The fix adds a check to skip attributes that are types (isinstance(attr, type)) in the _should_wrap_* functions for all four decorators: - ownership.py: _should_wrap_ownership_method - context.py: _should_wrap_madsci_context_method, _should_wrap_method - otel/tracing.py: _should_wrap_traced_method
Comprehensively update READMEs, AGENTS.md files, and guides to document: - EventClient context system for hierarchical logging - Structured logging best practices (kwargs over f-strings) - OpenTelemetry integration for distributed tracing - Ownership context for metadata tracking - Context decorators (@with_event_client, @event_client_class) Updated files include: - Root README.md and AGENTS.md with feature overview - madsci_client README.md and AGENTS.md with EventClient patterns - madsci_common README.md and AGENTS.md with context/OTEL docs - madsci_event_manager README.md and AGENTS.md - example_lab README.md with observability section - docs/guides/logging.md with decorators and OTEL sections - Regenerated API docs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces significant improvements to MADSci's event handling, logging, context management, and observability infrastructure. Key changes include:
OpenTelemetry Integration
madsci.common.otelmodule with bootstrap, tracing, and instrumentation utilitiesContext Management System
event_client_context()andget_event_client()@with_event_client,@event_client_class,@with_madsci_context@with_ownershipand@ownership_class@with_span,@traced_classEventClient Improvements
Example Lab Observability
Documentation Updates
Testing
Breaking Changes
None - all changes are backward compatible. Existing
EventClientusage continues to work.