Skip to content

Feature: Complete Event Manager Improvements - #219

Merged
RyanTheRobothead merged 105 commits into
unstablefrom
event_improvements
Feb 8, 2026
Merged

Feature: Complete Event Manager Improvements#219
RyanTheRobothead merged 105 commits into
unstablefrom
event_improvements

Conversation

@RyanTheRobothead

@RyanTheRobothead RyanTheRobothead commented Jan 30, 2026

Copy link
Copy Markdown
Member

Summary

This PR introduces significant improvements to MADSci's event handling, logging, context management, and observability infrastructure. Key changes include:

OpenTelemetry Integration

  • Full OTEL stack support with traces, metrics, and logs
  • New madsci.common.otel module with bootstrap, tracing, and instrumentation utilities
  • Automatic FastAPI and requests library instrumentation
  • Configurable per-manager via environment variables

Context Management System

  • Hierarchical logging context via event_client_context() and get_event_client()
  • Context decorators: @with_event_client, @event_client_class, @with_madsci_context
  • Ownership context tracking with @with_ownership and @ownership_class
  • Tracing decorators: @with_span, @traced_class

EventClient Improvements

  • Modern structlog-based API with context binding
  • OTEL integration for automatic trace correlation
  • Per-instance logger configuration
  • Metrics for event operations (send latency, buffer size, retries)

Example Lab Observability

  • Complete observability stack: Jaeger, Prometheus, Loki, Grafana
  • Pre-configured dashboards and datasources
  • OTEL collector with full pipeline configuration

Documentation Updates

  • Root README.md: Added observability to feature list, new Guides section
  • Root AGENTS.md: Added comprehensive logging, context, OTEL, and ownership sections
  • madsci_client README.md/AGENTS.md: EventClient structured logging, context binding, OTEL integration
  • madsci_common README.md/AGENTS.md: Context management, ownership, OpenTelemetry docs
  • madsci_event_manager README.md/AGENTS.md: Modern EventClient API, logging best practices
  • example_lab/README.md: Added observability stack section with quick start
  • docs/guides/logging.md: Extended with decorators and OTEL sections
  • example_lab/OBSERVABILITY.md: Complete observability stack setup guide
  • Regenerated API documentation

Testing

  • New test suites for EventClient context and OTEL integration
  • Context propagation tests across nested scopes
  • Decorator tests for ownership, tracing, and event client

Breaking Changes

None - all changes are backward compatible. Existing EventClient usage continues to work.

RyanTheRobothead and others added 12 commits January 1, 2026 13:00
Bumps ghcr.io/devcontainers-extra/features/ruff from 1.1.1 to 2.0.0.

---
updated-dependencies:
- dependency-name: ghcr.io/devcontainers-extra/features/ruff
  dependency-version: 2.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
…devcontainers-extra/features/ruff-2.0.0

Bump ghcr.io/devcontainers-extra/features/ruff from 1.1.1 to 2.0.0
@github-actions

github-actions Bot commented Jan 30, 2026

Copy link
Copy Markdown

Coverage report

Warning

The diff for this PR is too large to be retrieved from GitHub's API (maximum 300 files). Diff coverage is not available for this PR.

This PR does not seem to contain any modification to coverable code.

Mark Phase 1 complete and note Phase 2 progress/remaining work.
Expose OTEL status in /health output and add OTEL settings to ManagerSettings to drive manager-level tracing configuration.
Add domain spans for workflow execution, per-step execution, and node action dispatch to improve end-to-end trace visibility.
Document WorkcellManager domain spans (workflow.execute/workflow.step/node.action) and update acceptance criteria notes.
Wrap datapoint ingestion in a data.save span and update Phase 2 progress notes.
Add domain spans for resource lifecycle/query endpoints and for location creation, resource attachments, and transfer planning.
@RyanTheRobothead RyanTheRobothead self-assigned this Feb 7, 2026
@RyanTheRobothead RyanTheRobothead added enhancement New feature or request labels Feb 7, 2026
@RyanTheRobothead
RyanTheRobothead marked this pull request as ready for review February 7, 2026 00:01

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR delivers a broad observability overhaul across MADSci: introducing process-wide OpenTelemetry bootstrapping, a hierarchical EventClient context system, and a structured-logging migration backed by new tests and example-lab infrastructure.

Changes:

  • Added canonical OTEL bootstrap/helpers (madsci.common.otel) plus manager/client runtime integration, spans, and example-lab observability stack.
  • Introduced contextvars-based EventClient context propagation (middleware + client mixin integration) for hierarchical structured logging.
  • Added logging/audit tooling and expanded EventType usage, with extensive new tests and docs.

Reviewed changes

Copilot reviewed 105 out of 227 changed files in this pull request and generated 10 comments.

Show a summary per file
File Description
src/madsci_event_manager/madsci/event_manager/notifications.py Migrates email alert logs to structured format with event_type.
src/madsci_event_manager/madsci/event_manager/events_csv_exporter.py Adds CSV export logging (currently via stdlib logger).
src/madsci_data_manager/madsci/data_manager/data_server.py Adds OTEL spans around datastore endpoints and structured startup logs.
src/madsci_common/tests/test_manager_context.py Adds tests for manager + middleware context integration.
src/madsci_common/tests/test_manager_base_otel_runtime_reuse.py Verifies OTEL runtime reuse behavior in manager base.
src/madsci_common/tests/test_event_types_phase3.py Ensures EventType descriptions cover all enum members.
src/madsci_common/tests/test_event_type_coverage.py Adds EventType coverage/pattern tests + validates logging hook/docs exist.
src/madsci_common/pyproject.toml Adds OTEL API dependency and optional OTEL/instrumentation extras.
src/madsci_common/madsci/common/utils.py Replaces f-string logging with structured/printf-style logging.
src/madsci_common/madsci/common/types/manager_types.py Adds OTEL settings fields on ManagerSettings; adjusts ManagerType.
src/madsci_common/madsci/common/otel/requests_instrumentation.py Adds best-effort requests auto-instrumentation helper.
src/madsci_common/madsci/common/otel/propagation.py Adds helper to inject OTEL propagation headers.
src/madsci_common/madsci/common/otel/fastapi_instrumentation.py Adds best-effort FastAPI auto-instrumentation helper.
src/madsci_common/madsci/common/otel/init.py Exposes canonical OTEL bootstrap and helper APIs.
src/madsci_common/madsci/common/mongodb_version_checker.py Migrates logs to structured pattern w/ exc_info=True.
src/madsci_common/madsci/common/middleware.py Adds EventClientContextMiddleware + lazy context wrapper.
src/madsci_common/madsci/common/manager_base.py Adds manager OTEL bootstrap, span helper, middleware wiring, structured logs.
src/madsci_common/madsci/common/context.py Implements EventClient context APIs via contextvars.
src/madsci_common/madsci/common/backup_tools/postgres_backup.py Migrates backup logs to structured EventType-based logging.
src/madsci_common/madsci/common/backup_tools/backup_validator.py Migrates validator logs to structured EventType-based logging.
src/madsci_common/madsci/common/backup_tools/backup_manager.py Migrates backup manager logs to structured EventType-based logging.
src/madsci_common/madsci/common/init.py Installs rich traceback handler globally on import.
src/madsci_client/tests/test_event_client_otel_metrics_subprocess.py Adds subprocess test for deterministic OTEL metrics assertions.
src/madsci_client/tests/test_event_client_otel.py Adds EventClient OTEL integration tests (trace context + instruments).
src/madsci_client/tests/test_event_client_context.py Adds comprehensive tests for EventClient context API + async propagation.
src/madsci_client/tests/test_client_context_integration.py Adds tests ensuring clients/mixin participate in context inheritance.
src/madsci_client/pyproject.toml Adds OTEL API + structlog dependency; OTLP exporter as optional extra.
src/madsci_client/madsci/client/workcell_client.py Switches to context-aware logging client creation + structured error logs.
src/madsci_client/madsci/client/structlog_config.py Adds per-instance structlog configuration with OTEL/hierarchy processors.
src/madsci_client/madsci/client/otel_processors.py Adds structlog processors injecting OTEL trace context into logs.
src/madsci_client/madsci/client/node/rest_node_client.py Adds context-aware logger creation + structured error logs.
src/madsci_client/madsci/client/location_client.py Uses context-aware logger creation; adds warning event_type field.
src/madsci_client/madsci/client/data_client.py Adds EventType to warnings in data client operations.
src/madsci_client/madsci/client/client_mixin.py Makes mixin EventClient resolution context-aware + adds explicit teardown.
src/madsci_client/README.md Documents context-based logging usage patterns for clients.
scripts/precommit_check_logging_patterns.py Adds pre-commit hook to ratchet structured logging patterns.
ruff.toml Adds excludes/ignores for scripts + markdown.
pyrightconfig.json Adds pyright config with relaxed reporting settings.
pyproject.toml Adds structlog/colorama dependencies; adds OTEL + pytest-asyncio to dev deps.
myst.yml Updates docs TOC paths and adds docs/madsci_powered section.
madsci-entrypoint.sh Ensures venv env vars are set during container entrypoint.
examples/otel_demo.py Adds OTEL demo script using canonical bootstrap and trace correlation examples.
examples/README_OTEL_POC.md Adds OTEL PoC README documentation.
example_lab/otel/prometheus.yaml Adds Prometheus config for OTEL demo stack.
example_lab/otel/loki.yaml Adds Loki config for OTEL demo stack.
example_lab/otel/grafana/provisioning/datasources/datasources.yaml Provisions Grafana datasources for Jaeger/Prometheus/Loki.
example_lab/otel/grafana/provisioning/dashboards/dashboards.yaml Provisions Grafana dashboard provider.
example_lab/otel-collector.yaml Adds basic OTEL collector config (debug exporter).
example_lab/otel-collector-full.yaml Adds full OTEL collector config (Jaeger/Prom/Loki exporters).
example_lab/notebooks/experiment_notebook.ipynb Updates notebook code style + kernel metadata.
example_lab/example_transfer.py Wraps example transfer script in EventClient context for hierarchical logging.
example_lab/example_app.info.yaml Updates example app capabilities + rate limiting/uvicorn configuration schema.
example_lab/README.md Adds link to real module examples and expands requirements wording.
example_lab/OBSERVABILITY.md Adds end-to-end guide for the observability stack.
docs/madsci_powered/Modules.md Adds list of integrated modules/devices.
docs/madsci_powered/Labs_and_Workcells.md Adds list of MADSci-powered labs/workcells.
docs/madsci_powered/Experiments.md Adds list of MADSci-enabled experiments.
docs/dev/opentelemetry_integration_plan/README.md Adds OTEL integration plan index.
docs/dev/opentelemetry_integration_plan/14_phase_4_example_lab_infra.md Documents example lab OTEL infra implementation details.
docs/dev/opentelemetry_integration_plan/10_phase_0_dependency_setup.md Documents OTEL dependency strategy and acceptance criteria.
docs/dev/opentelemetry_integration_plan/02_current_state_assessment.md Documents current OTEL/logging state assessment.
docs/dev/opentelemetry_integration_plan/01_overview.md Documents OTEL plan overview and measurable goals.
docs/dev/opentelemetry_integration_plan/00_principles.md Documents OTEL design principles (cardinality, precedence, shutdown).
docs/dev/logging_guidelines.md Adds structured logging and context usage guidelines.
docs/dev/event_client_context_plan/README.md Adds context system plan + quick reference.
docs/dev/event_client_context_plan/00_principles.md Documents context system design decisions and naming conventions.
docs/api/madsci/experiment_application/index.md Updates generated API docs output path/content.
docs/api/madsci/experiment_application/experiment_application.md Updates generated API docs output path/content.
docs/api/madsci/common/types/resource_types/index.md Updates generated API docs for resource types.
docs/api/madsci/client/resource_client.md Updates generated API docs for resource client signatures.
docs/api/madsci/client/index.md Updates generated API docs for client index signatures.
devbox.json Adds devbox environment configuration (python/uv/pdm).
compose.yaml Includes compose.otel.yaml and makes optional .env env_file.
compose.otel.yaml Adds OTEL observability stack services (collector/jaeger/prom/loki/grafana).
README.md Adds JOSS badge.
Dockerfile.dashboard Adds venv verification steps during dashboard image build.
Dockerfile Switches to PDM+uv venv install flow and installs OTEL extras.
.pre-commit-config.yaml Adds local pre-commit hook for logging pattern enforcement.
.justfile Writes generated pdoc output into docs/api/.
.github/workflows/gh_pages.yml Updates pdoc output path to docs/api/.
.github/workflows/e2e_tests.yml Increases compose startup wait to 60s for e2e.
.envrc Adds direnv+devbox integration.
.env.example Adds OTEL + retention/log-rotation configuration examples.
.devcontainer/devcontainer.json Updates ruff devcontainer feature version.
Comments suppressed due to low confidence (1)

src/madsci_common/madsci/common/types/manager_types.py:1

  • This change removes the previous ManagerType._missing_ case-insensitive coercion behavior. If any YAML/env/config inputs relied on e.g. Event_Manager / EVENT_MANAGER / EventManager, they will now fail validation. If backward compatibility is a goal, reintroduce case-insensitive parsing (either via _missing_ or a dedicated pydantic validator that normalizes input before enum parsing).
"""Types used primarily by MADSci Managers."""

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/madsci_event_manager/madsci/event_manager/events_csv_exporter.py Outdated
Comment thread src/madsci_common/madsci/common/__init__.py Outdated
Comment thread src/madsci_client/madsci/client/location_client.py Outdated
Comment thread src/madsci_client/madsci/client/data_client.py Outdated
Comment thread src/madsci_common/madsci/common/middleware.py
Comment thread src/madsci_common/madsci/common/middleware.py
Comment thread src/madsci_common/madsci/common/middleware.py
Comment thread src/madsci_common/madsci/common/context.py Outdated
Comment thread docs/api/madsci/experiment_application/index.md Outdated
Comment thread docs/madsci_powered/Modules.md Outdated
Add diagnostic steps to help debug CI failures:
- Check service health endpoints before running tests
- Show docker compose status and lab_manager logs pre-test
- Show detailed service logs on test failure
Root cause: Docker Compose builds images in parallel, causing the
dashboard image to pull the old published base image from GHCR instead
of using the locally built one. This meant structlog and pytz were
missing.

Fixes:
- Build base madsci image first before docker compose up
- Add pytz to event_manager dependencies (was imported but not declared)
- Update pdm.lock with pytz dependency
Changes:
- e2e_tests now triggers after Docker Image workflow completes
- Pulls pre-built images from GHCR instead of rebuilding
- Adds docker workflow trigger for pull_request events
- Properly handles PR tags (pr-<number>) and branch tags
- Checkouts the correct commit for workflow_run events
- Adds manual trigger option to build locally if needed

Benefits:
- Faster e2e tests (no image rebuild)
- Ensures e2e tests use same images that will be published
- Prevents build order issues between base and dashboard images
@RyanTheRobothead RyanTheRobothead changed the title Feature: Complete Event Manager Improvements (Phases 1-5) Feature: Complete Event Manager Improvements Feb 7, 2026
- Add concurrency group to docker workflow to cancel duplicate runs
- When both push and pull_request events fire for the same ref, only
  one build will complete (the newer one cancels the older)
- Update e2e_tests to reference renamed workflow 'Build Docker Images'
Use github.head_ref (for PRs) or github.ref_name (for push) so that
both events for the same branch resolve to the same concurrency group.

Example: PR from 'feature-x' and push to 'feature-x' both get
group 'docker-feature-x', ensuring only one build runs.
Instead of using concurrency with cancel-in-progress (which causes
cancelled workflows to show as failed checks), simply limit push
triggers to main branch and tags.

- PRs trigger via pull_request event (covers all feature branches)
- Merges to main trigger via push event
- Tags trigger via push event

This eliminates duplicate runs without any cancelled/failed checks.
Documentation:
- Create docs/guides/logging.md combining logging guidelines and context
  migration guide into a single production-ready document
- Add Guides section to myst.yml table of contents
- Add example_lab/OBSERVABILITY.md to TOC (already existed)
- Remove docs/dev/ planning documents (19 files)

Code cleanup:
- Remove phase references from EventType comments
- Rename test_event_types_phase3.py to test_event_type_descriptions.py
- Update test docstring to remove phase reference

All valuable documentation from the planning docs has been extracted
into production docs. The OBSERVABILITY.md in example_lab already
contained the complete user guide for the OTEL stack.
…ement

This commit introduces class and function decorators to simplify context
management across MADSci components:

## New Features

### Ownership Context Decorators (ownership.py)
- @with_ownership: Function decorator for ownership context
- @ownership_class: Class decorator that wraps all public methods
- get_ownership_overrides() method support for dynamic ownership

### OpenTelemetry Tracing Decorators (otel/tracing.py)
- span_context(): Context manager for creating spans
- @with_span: Function decorator for tracing
- @traced_class: Class decorator for automatic method tracing
- Utility functions: get_tracer(), get_current_span(), set_span_error(), etc.

### MadsciContext Decorators (context.py)
- @with_madsci_context: Function decorator for context management
- @madsci_context_class: Class decorator for automatic context wrapping

## Refactoring

### LocationManager
- Applied @ownership_class decorator
- Removed 10 manual `with ownership_context():` blocks

### WorkcellEngine
- Replaced manual tracer with span_context() from tracing module
- Removed self._tracer initialization

### ResourceManager
- Applied @ownership_class decorator with get_ownership_overrides()
- Removed _setup_ownership() method
- Removed HTTP middleware for ownership context

## Tests
- 91 new decorator tests (ownership, context, tracing, stacking)
- All existing tests pass (215 resource manager, 107 workcell, 101 location)
The ownership_class, madsci_context_class, with_event_client_class, and
traced_class decorators were wrapping class attributes that are types
(like DEFINITION_CLASS and SETTINGS_CLASS) as if they were instance
methods. This caused TypeError when trying to instantiate these classes.

The fix adds a check to skip attributes that are types (isinstance(attr, type))
in the _should_wrap_* functions for all four decorators:
- ownership.py: _should_wrap_ownership_method
- context.py: _should_wrap_madsci_context_method, _should_wrap_method
- otel/tracing.py: _should_wrap_traced_method
Comprehensively update READMEs, AGENTS.md files, and guides to document:

- EventClient context system for hierarchical logging
- Structured logging best practices (kwargs over f-strings)
- OpenTelemetry integration for distributed tracing
- Ownership context for metadata tracking
- Context decorators (@with_event_client, @event_client_class)

Updated files include:
- Root README.md and AGENTS.md with feature overview
- madsci_client README.md and AGENTS.md with EventClient patterns
- madsci_common README.md and AGENTS.md with context/OTEL docs
- madsci_event_manager README.md and AGENTS.md
- example_lab README.md with observability section
- docs/guides/logging.md with decorators and OTEL sections
- Regenerated API docs
@RyanTheRobothead
RyanTheRobothead merged commit 9615fd2 into unstable Feb 8, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

2 participants