Location Manager: template-based locations, MongoDB persistence, and declarative node template registration - #228
Merged
RyanTheRobothead merged 32 commits intoMar 20, 2026
Conversation
RyanTheRobothead
marked this pull request as draft
February 23, 2026 16:01
RyanTheRobothead
marked this pull request as ready for review
March 5, 2026 17:19
- Fix LocationReservation.check() referencing removed start/end fields (now uses created/expires) - Fix indentation bug in get_location_resources() making resource hierarchy query unreachable - Fix Path.open() misuse in initialize() and add model_dump() for YAML serialization - Fix dict indexing on Pydantic model in update_location() error message - Fix add_location endpoint silently returning None on duplicate (now raises 409) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Update LocationClient methods (delete_location, attach_resource,
detach_resource, get_location_resources) to use location_name
matching the server's name-based routing
- Route get_location(location_id) through query endpoint to avoid
path ambiguity with /location/{location_name}
- Remove redundant /location/id/{location_id} endpoint (covered by
GET /location?location_id=...)
- Fix import_locations() to handle both list and dict YAML, close file handle
- Re-add reservation field to Location model for future reservation support
- Update Location docstring from "Definition" to "A location in the lab"
- Fix stale "by ID" docstrings across server, state handler, and client
- Update workcell_actions.py to use location_name for get_location_resources
- Update client and server tests to match new API surface
- Fix location manager initialization to load locations from file on
startup (file is the source of truth for location definitions)
- Add depends_on to notebook_validator compose service (all managers + nodes)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move persistent location data from Redis (ephemeral cache) to MongoDB (document storage), keeping Redis for transient state only (locks, change counters). This fixes the architectural mismatch where long-lived configuration data was stored in an ephemeral store. Key changes: - LocationStateHandler: MongoDB for CRUD, Redis for transient state - LocationManager: MongoDB version checking, seed file loading, bulk import/export endpoints, auto-migration from 0.7.1 Redis format - LocationClient: import_locations POSTs to /locations/import endpoint, new export_locations and close methods - LocationMigrator: one-time 0.7.1 Redis → MongoDB migration tool - Settings: document_db_url, database_name, seed_locations_file (renamed from locations_file_path with backward-compat alias) - Health: reports both document_db_connected and redis_connected - LocalRunner: passes InMemoryMongoHandler to location manager - 57 new tests (151 total location tests, 2723 full suite) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…te registration - Add NodeResourceTemplateDefinition, NodeRepresentationTemplateDefinition, and NodeLocationTemplateDefinition types to node_types.py - Add template_handler() method to AbstractNode that registers templates from ClassVar lists with per-template error isolation - Call template_handler() before startup_handler() in _startup() so templates are available for create_resource_from_template() calls - Move template registrations from startup_handler() to class-level declarations in liquidhandler, robotarm, and platereader example nodes - Rename LocationRepresentationTemplate.schema to schema_def (with backward-compatible validation_alias) to avoid shadowing Python builtin - Update LocationClient.init_representation_template() to accept both schema (deprecated) and schema_def parameters - Add 17 tests covering registration, error isolation, call order, error logging, and definition type validation - Use structured logging (kwargs) instead of f-strings in all new log calls Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add URL-safe location name validator, fix TOCTOU races in state handler add methods with unique indexes + DuplicateKeyError, move inline imports to top-level, use walk-up discovery for seed file paths, replace app._manager with app.state.manager, guard race condition fallbacks, extract shared reconciliation helper, fix stale test kwargs, and enhance InMemoryCollection to enforce unique index constraints. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Extract a shared ensure_schema_indexes() function that idempotently creates all indexes defined in schema.json, and call it from both startup auto-init (validate_or_fail) and migration tool (apply_schema_migrations). This ensures fresh databases get all data collection indexes without needing the migration CLI. - Add ensure_schema_indexes() free function and MongoDBVersionChecker method - Add list_indexes() to InMemoryCollection for pymongo interface compat - Call ensure_schema_indexes() from validate_or_fail() on fresh DB auto-init - Refactor MongoDBMigrator.apply_schema_migrations() to use shared function - Call ensure_schema_indexes() from LocationManager.initialize() for test/prod - Add 7 new tests covering index creation, idempotency, and error handling Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
# Conflicts: # examples/example_lab/compose.yaml # src/madsci_common/madsci/common/document_db_migration_tool.py # src/madsci_common/madsci/common/document_db_version_checker.py # src/madsci_common/madsci/common/local_backends/local_runner.py # src/madsci_common/tests/test_mongodb_version_checker.py
…r-neutral terms Systematically update all remaining references to proprietary database products (MongoDB, Redis, MinIO) with vendor-neutral terminology (document database, cache, object storage) or FOSS product names (FerretDB, Valkey, SeaweedFS) throughout comments, docstrings, config fields, documentation, and test files. Breaking changes (beta): - env_prefix MONGODB_ → DOCUMENT_DB_ (DocumentDBBackupSettings) - env_prefix MONGODB_MIGRATION_ → DOCUMENT_DB_MIGRATION_ (DocumentDBMigrationSettings) - Fields redis_host/port/password → cache_host/port/password (WorkcellManagerSettings, LocationManagerSettings) - Fields redis_connected → cache_connected (health models) - Docker types REDIS_PORT → CACHE_PORT - Backup metadata backup_type value "mongodb" → "document_db" All old field names remain accepted via validation_alias for backward compatibility. 2889 tests passing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Continues the FOSS terminology audit by renaming the cache handler abstraction layer from Redis-specific names to generic cache names, consistent with the earlier config field renames (redis_host → cache_host). Renames: - redis_handler.py → cache_handler.py - RedisHandler → CacheHandler (ABC) - PyRedisHandler → PyCacheHandler (real implementation) - InMemoryRedisHandler → InMemoryCacheHandler (test implementation) - redis_handler param/attr → cache_handler/_cache_handler Wire protocol references (redis.Redis, pottery.RedisDict, InMemoryRedisClient) are kept as-is since they refer to the underlying client libraries. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… manager index name conflicts The version checker's validate_or_fail() only auto-initialized completely empty databases (current=None), but refused to start when collections existed without version tracking (current=0.0.0). This caused managers to crash-loop when FerretDB volumes persisted across restarts. Extended auto-initialization to also handle the 0.0.0 case, since both mean no prior version tracking existed. Also fixed LocationStateHandler creating indexes with names that conflicted with schema.json definitions (template_name_unique vs repr_template_name_unique / loc_template_name_unique), causing FerretDB IndexOptionsConflict errors. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Use timezone-aware datetime.now(timezone.utc) in location types and server - Move state_change_marker and shutdown from class to instance attributes - Add efficient count methods to LocationStateHandler for health endpoint - Defer reconciliation in init_representation_template to background loop - Fix representations guard to handle both None (legacy) and empty dict - Remove bogus experiment_id_1 unique index from experiment manager schema (experiment_id is aliased to _id by to_mongo(), so the index on the non-existent field caused DuplicateKeyError on second insert) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… optimize reconciliation - Fix LocationReservation.expires field title to "Expires Datetime" - Simplify LocationReservation.check() double-negation to clear positive logic - Remove dead None check in remove_representation (dict default_factory) - Fix plan_transfer docstring parameter names to match actual args - Expose _document_handler via public property on LocationStateHandler - Add get_unresolved_locations() with targeted $or query for reconciliation - Rename --mongo-url CLI flag to --document-db-url in migration tool - Add $or/$and logical operator support to in-memory collection query matcher Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This was referenced Mar 20, 2026
RyanTheRobothead
added a commit
that referenced
this pull request
Mar 31, 2026
- Complete CHANGELOG for v0.8.0 with all 9 merged PRs (#228, #235, #242, #255–#260), properly categorized under Added/Changed/Fixed - Fix broken ActionHandler import in module/basic and node/basic templates (replaced with @action decorator pattern) - Fix self.node_definition → self.node_info in 5 module templates (device, instrument, camera, liquid_handler, robot_arm) and their READMEs - Modernize self.logger.log() → self.event_client.info() in same 5 templates - Fix stale infrastructure references in example lab README (MongoDB→FerretDB, MinIO→SeaweedFS, wrong ports, deprecated /definition endpoint) - Update node module README to remove deprecated --node_definition reference Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Major architectural overhaul of the Location Manager to support template-based location definitions, MongoDB persistence, and declarative template registration from nodes.
Location Manager: MongoDB + Redis Dual-Handler
schema.json+schema_versionscollection) to prevent silent data corruptionlocation_migration.py) for upgrading from v0.7.1Template System for Locations
robotarm_deck_accesswith default gripper config and required position overrides)deck_controller,transfer_arm) to representation templatesinitendpoints for both template typesPOST /location/from_template: Instantiate locations from templates with node bindings and representation overridesDeclarative Node Template Registration
template_handler()lifecycle hook inAbstractNode, called beforestartup_handler()resource_templates,location_representation_templates,location_templates)Location Client Enhancements
import_locations()/export_locations()for bulk operationsLocationClientConfig)Example Lab Updates
locations.yamlseed file demonstrating the template-based formatUI: Template Panels
LocationTemplatesPanel,RepresentationTemplatesPanel, andResourceTemplatesPanelcomponents in the Squid DashboardWorkcell Manager
get_location_by_name()instead of manual search over all locationsTesting
Test plan
pytestto verify all tests passjust upand verify location manager initializes with MongoDB persistence🤖 Generated with Claude Code