Skip to content

Location Manager: template-based locations, MongoDB persistence, and declarative node template registration - #228

Merged
RyanTheRobothead merged 32 commits into
unstablefrom
88-feature-node-and-location-reservations
Mar 20, 2026
Merged

Location Manager: template-based locations, MongoDB persistence, and declarative node template registration#228
RyanTheRobothead merged 32 commits into
unstablefrom
88-feature-node-and-location-reservations

Conversation

@tginsbu1

@tginsbu1 tginsbu1 commented Feb 20, 2026

Copy link
Copy Markdown
Contributor

Summary

Major architectural overhaul of the Location Manager to support template-based location definitions, MongoDB persistence, and declarative template registration from nodes.

Location Manager: MongoDB + Redis Dual-Handler

  • Migrated location persistence from Redis-only to MongoDB (authoritative CRUD) + Redis (transient state: locks, change counters)
  • Added schema versioning (schema.json + schema_versions collection) to prevent silent data corruption
  • Auto-migration at startup: old Redis data → MongoDB (one-time, transparent)
  • Standalone migration CLI (location_migration.py) for upgrading from v0.7.1

Template System for Locations

  • Representation Templates: Named, versioned blueprints for node-specific location representations (e.g., robotarm_deck_access with default gripper config and required position overrides)
  • Location Templates: Named, versioned blueprints for creating locations that map abstract roles (e.g., deck_controller, transfer_arm) to representation templates
  • Full CRUD + idempotent init endpoints for both template types
  • POST /location/from_template: Instantiate locations from templates with node bindings and representation overrides
  • Lazy resolution: Locations can be created before all templates exist; reconciliation fills in missing data when templates register later

Declarative Node Template Registration

  • New template_handler() lifecycle hook in AbstractNode, called before startup_handler()
  • Nodes declare templates as class variables (resource_templates, location_representation_templates, location_templates)
  • Templates registered automatically at startup with exponential-backoff retry (tolerates manager unavailability)
  • Single template failure does not prevent node startup

Location Client Enhancements

  • New methods for representation template, location template, and template-based location CRUD
  • import_locations() / export_locations() for bulk operations
  • Configurable startup retry with exponential backoff (LocationClientConfig)
  • Locations now primarily queried/deleted by name instead of ID

Example Lab Updates

  • New locations.yaml seed file demonstrating the template-based format
  • Robot arm, liquid handler, and plate reader modules refactored to use declarative class-variable templates
  • Docker Compose updated with MongoDB dependency for location manager and explicit service ordering for nodes

UI: Template Panels

  • New LocationTemplatesPanel, RepresentationTemplatesPanel, and ResourceTemplatesPanel components in the Squid Dashboard
  • Store integration for template data binding

Workcell Manager

  • Minor refactor: location lookups use get_location_by_name() instead of manual search over all locations

Testing

  • 6 new test files: migration, template server CRUD, reconciliation, schema validation, seed format parsing, node template handler
  • 3 significantly expanded test files: location client, server, and state handler
  • ~2600+ lines of new/updated tests covering all new functionality

Test plan

  • Run pytest to verify all tests pass
  • Start example lab with just up and verify location manager initializes with MongoDB persistence
  • Verify template registration from example node modules (robot arm, liquid handler, plate reader)
  • Test lazy resolution: start nodes before location manager, confirm reconciliation fills in templates
  • Verify UI template panels display registered templates in dashboard
  • Test migration path from Redis-only installation

🤖 Generated with Claude Code

@tginsbu1 tginsbu1 linked an issue Feb 20, 2026 that may be closed by this pull request
@RyanTheRobothead
RyanTheRobothead marked this pull request as draft February 23, 2026 16:01
@RyanTheRobothead
RyanTheRobothead marked this pull request as ready for review March 5, 2026 17:19
@tginsbu1
tginsbu1 changed the base branch from main to unstable March 12, 2026 21:49
@github-actions

github-actions Bot commented Mar 12, 2026

Copy link
Copy Markdown

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  src/madsci_client/madsci/client
  location_client.py 107-130, 186-188, 252-253, 286-302, 344-351, 359-366, 374-382, 405-428, 436-443, 451-458, 464-471, 479-487, 507-530, 538-545, 561-578, 582-583, 603-611, 639-641, 703-712, 811
  src/madsci_common/madsci/common
  document_db_migration_tool.py 147-155
  document_db_version_checker.py 59-60, 81-83, 87, 102-103, 380-387, 495, 508
  validators.py
  src/madsci_common/madsci/common/backup_tools
  document_db_backup.py 162-167, 391, 431, 540, 544-546, 657-659, 696, 700-702
  document_db_cli.py 72, 282
  src/madsci_common/madsci/common/db_handlers
  cache_handler.py 143-144
  src/madsci_common/madsci/common/local_backends
  inmemory_collection.py 164, 171, 451, 461, 463-464
  local_runner.py 166-178
  src/madsci_common/madsci/common/types
  backup_types.py 90
  client_types.py
  document_db_migration_types.py
  location_types.py 334-337, 582
  node_types.py
  workcell_types.py
  src/madsci_location_manager/madsci/location_manager
  location_migration.py 57, 60-61, 80-82, 99-100, 123-124, 128-129, 162, 214-215, 222-266, 270
  location_server.py 69, 97-137, 222-249, 268, 275, 285, 294-299, 306-307, 324-326, 334-335, 369-370, 382-383, 400-401, 449, 512, 518, 535-541, 547-549, 564, 573, 652, 717, 790-792, 865-873, 948-956, 1045, 1072-1074, 1126-1138, 1152, 1158, 1164, 1177, 1183, 1189-1190, 1204-1206, 1210-1219, 1312-1316, 1333-1357, 1365-1371
  location_state_handler.py 60, 73-76, 86, 119, 190-191, 203, 211-212, 236-237, 275, 309-310, 318-319, 328, 343, 383-384, 392-393, 402, 419
  transfer_planner.py
  src/madsci_node_module/madsci/node_module
  abstract_node_module.py
  src/madsci_workcell_manager/madsci/workcell_manager
  condition_checks.py 54, 59
  state_handler.py 50, 70-73, 211-214, 379-384, 388-390, 455, 459-460
  workcell_actions.py
  workcell_server.py 79, 143, 163-165, 272, 284-288
  src/madsci_workcell_manager/madsci/workcell_manager/schedulers
  default_scheduler.py 96-114
Project Total  

The report is truncated to 25 files out of 26. To see the full report, please visit the workflow summary page.

This report was generated by python-coverage-comment-action

RyanTheRobothead and others added 11 commits March 13, 2026 15:34
- Fix LocationReservation.check() referencing removed start/end fields (now uses created/expires)
- Fix indentation bug in get_location_resources() making resource hierarchy query unreachable
- Fix Path.open() misuse in initialize() and add model_dump() for YAML serialization
- Fix dict indexing on Pydantic model in update_location() error message
- Fix add_location endpoint silently returning None on duplicate (now raises 409)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Update LocationClient methods (delete_location, attach_resource,
  detach_resource, get_location_resources) to use location_name
  matching the server's name-based routing
- Route get_location(location_id) through query endpoint to avoid
  path ambiguity with /location/{location_name}
- Remove redundant /location/id/{location_id} endpoint (covered by
  GET /location?location_id=...)
- Fix import_locations() to handle both list and dict YAML, close file handle
- Re-add reservation field to Location model for future reservation support
- Update Location docstring from "Definition" to "A location in the lab"
- Fix stale "by ID" docstrings across server, state handler, and client
- Update workcell_actions.py to use location_name for get_location_resources
- Update client and server tests to match new API surface
- Fix location manager initialization to load locations from file on
  startup (file is the source of truth for location definitions)
- Add depends_on to notebook_validator compose service (all managers + nodes)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move persistent location data from Redis (ephemeral cache) to MongoDB
(document storage), keeping Redis for transient state only (locks, change
counters). This fixes the architectural mismatch where long-lived
configuration data was stored in an ephemeral store.

Key changes:
- LocationStateHandler: MongoDB for CRUD, Redis for transient state
- LocationManager: MongoDB version checking, seed file loading, bulk
  import/export endpoints, auto-migration from 0.7.1 Redis format
- LocationClient: import_locations POSTs to /locations/import endpoint,
  new export_locations and close methods
- LocationMigrator: one-time 0.7.1 Redis → MongoDB migration tool
- Settings: document_db_url, database_name, seed_locations_file (renamed
  from locations_file_path with backward-compat alias)
- Health: reports both document_db_connected and redis_connected
- LocalRunner: passes InMemoryMongoHandler to location manager
- 57 new tests (151 total location tests, 2723 full suite)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…te registration

- Add NodeResourceTemplateDefinition, NodeRepresentationTemplateDefinition,
  and NodeLocationTemplateDefinition types to node_types.py
- Add template_handler() method to AbstractNode that registers templates
  from ClassVar lists with per-template error isolation
- Call template_handler() before startup_handler() in _startup() so
  templates are available for create_resource_from_template() calls
- Move template registrations from startup_handler() to class-level
  declarations in liquidhandler, robotarm, and platereader example nodes
- Rename LocationRepresentationTemplate.schema to schema_def (with
  backward-compatible validation_alias) to avoid shadowing Python builtin
- Update LocationClient.init_representation_template() to accept both
  schema (deprecated) and schema_def parameters
- Add 17 tests covering registration, error isolation, call order,
  error logging, and definition type validation
- Use structured logging (kwargs) instead of f-strings in all new log calls

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@RyanTheRobothead RyanTheRobothead changed the title location update Location Manager: template-based locations, MongoDB persistence, and declarative node template registration Mar 16, 2026
Add URL-safe location name validator, fix TOCTOU races in state handler
add methods with unique indexes + DuplicateKeyError, move inline imports
to top-level, use walk-up discovery for seed file paths, replace
app._manager with app.state.manager, guard race condition fallbacks,
extract shared reconciliation helper, fix stale test kwargs, and enhance
InMemoryCollection to enforce unique index constraints.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RyanTheRobothead and others added 7 commits March 17, 2026 14:40
Extract a shared ensure_schema_indexes() function that idempotently creates
all indexes defined in schema.json, and call it from both startup auto-init
(validate_or_fail) and migration tool (apply_schema_migrations). This ensures
fresh databases get all data collection indexes without needing the migration
CLI.

- Add ensure_schema_indexes() free function and MongoDBVersionChecker method
- Add list_indexes() to InMemoryCollection for pymongo interface compat
- Call ensure_schema_indexes() from validate_or_fail() on fresh DB auto-init
- Refactor MongoDBMigrator.apply_schema_migrations() to use shared function
- Call ensure_schema_indexes() from LocationManager.initialize() for test/prod
- Add 7 new tests covering index creation, idempotency, and error handling

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
# Conflicts:
#	examples/example_lab/compose.yaml
#	src/madsci_common/madsci/common/document_db_migration_tool.py
#	src/madsci_common/madsci/common/document_db_version_checker.py
#	src/madsci_common/madsci/common/local_backends/local_runner.py
#	src/madsci_common/tests/test_mongodb_version_checker.py
…r-neutral terms

Systematically update all remaining references to proprietary database
products (MongoDB, Redis, MinIO) with vendor-neutral terminology
(document database, cache, object storage) or FOSS product names
(FerretDB, Valkey, SeaweedFS) throughout comments, docstrings, config
fields, documentation, and test files.

Breaking changes (beta):
- env_prefix MONGODB_ → DOCUMENT_DB_ (DocumentDBBackupSettings)
- env_prefix MONGODB_MIGRATION_ → DOCUMENT_DB_MIGRATION_ (DocumentDBMigrationSettings)
- Fields redis_host/port/password → cache_host/port/password (WorkcellManagerSettings, LocationManagerSettings)
- Fields redis_connected → cache_connected (health models)
- Docker types REDIS_PORT → CACHE_PORT
- Backup metadata backup_type value "mongodb" → "document_db"

All old field names remain accepted via validation_alias for backward
compatibility. 2889 tests passing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Continues the FOSS terminology audit by renaming the cache handler
abstraction layer from Redis-specific names to generic cache names,
consistent with the earlier config field renames (redis_host → cache_host).

Renames:
- redis_handler.py → cache_handler.py
- RedisHandler → CacheHandler (ABC)
- PyRedisHandler → PyCacheHandler (real implementation)
- InMemoryRedisHandler → InMemoryCacheHandler (test implementation)
- redis_handler param/attr → cache_handler/_cache_handler

Wire protocol references (redis.Redis, pottery.RedisDict, InMemoryRedisClient)
are kept as-is since they refer to the underlying client libraries.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… manager index name conflicts

The version checker's validate_or_fail() only auto-initialized completely empty
databases (current=None), but refused to start when collections existed without
version tracking (current=0.0.0). This caused managers to crash-loop when
FerretDB volumes persisted across restarts. Extended auto-initialization to also
handle the 0.0.0 case, since both mean no prior version tracking existed.

Also fixed LocationStateHandler creating indexes with names that conflicted with
schema.json definitions (template_name_unique vs repr_template_name_unique /
loc_template_name_unique), causing FerretDB IndexOptionsConflict errors.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Use timezone-aware datetime.now(timezone.utc) in location types and server
- Move state_change_marker and shutdown from class to instance attributes
- Add efficient count methods to LocationStateHandler for health endpoint
- Defer reconciliation in init_representation_template to background loop
- Fix representations guard to handle both None (legacy) and empty dict
- Remove bogus experiment_id_1 unique index from experiment manager schema
  (experiment_id is aliased to _id by to_mongo(), so the index on the
  non-existent field caused DuplicateKeyError on second insert)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@RyanTheRobothead RyanTheRobothead mentioned this pull request Mar 19, 2026
6 tasks
RyanTheRobothead and others added 2 commits March 20, 2026 10:41
… optimize reconciliation

- Fix LocationReservation.expires field title to "Expires Datetime"
- Simplify LocationReservation.check() double-negation to clear positive logic
- Remove dead None check in remove_representation (dict default_factory)
- Fix plan_transfer docstring parameter names to match actual args
- Expose _document_handler via public property on LocationStateHandler
- Add get_unresolved_locations() with targeted $or query for reconciliation
- Rename --mongo-url CLI flag to --document-db-url in migration tool
- Add $or/$and logical operator support to in-memory collection query matcher

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@RyanTheRobothead
RyanTheRobothead merged commit 928988c into unstable Mar 20, 2026
3 checks passed
RyanTheRobothead added a commit that referenced this pull request Mar 31, 2026
- Complete CHANGELOG for v0.8.0 with all 9 merged PRs (#228, #235, #242,
  #255#260), properly categorized under Added/Changed/Fixed
- Fix broken ActionHandler import in module/basic and node/basic templates
  (replaced with @action decorator pattern)
- Fix self.node_definition → self.node_info in 5 module templates (device,
  instrument, camera, liquid_handler, robot_arm) and their READMEs
- Modernize self.logger.log() → self.event_client.info() in same 5 templates
- Fix stale infrastructure references in example lab README (MongoDB→FerretDB,
  MinIO→SeaweedFS, wrong ports, deprecated /definition endpoint)
- Update node module README to remove deprecated --node_definition reference

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@RyanTheRobothead
RyanTheRobothead deleted the 88-feature-node-and-location-reservations branch April 9, 2026 19:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants