Repository navigation
feat(components): prototype XMC components system with GitOps, CMS, and Spanner SOT - #456
VLukiantsev wants to merge 2 commits into
Conversation
…nd Spanner SOT Implements the foundational prototype for the XMC Components System based on the AI Components Design Document (Option C: Hybrid Spanner SOT with GKE Informers). Key Capabilities: - ComponentManagerService (CMS) Protobuf gRPC definitions and server implementation. - Cloud Spanner migration (010 up/down) with interleaved dependencies and bit-reversed experiment snapshot locking (T0). - GitOps Bridge Controller (cmd/xmc-bridge) for reconciling Component CRDs, dynamic health probing, and 1P workload management. - 3-pass static manifest linter (xmc-linter) for syntax, DAG cycle detection, and semver compatibility matrix enforcement. - Initial component migrations for ClickHouse (3P wrapper), Redpanda (3P wrapper), and Ingestion Gateway (1P native). TAG=agy CONV=59b0a148-0a1a-4bd2-a98a-6f0c5c7e8e1a
There was a problem hiding this comment.
Code Review
This pull request introduces the XMC Components System, which includes a gRPC control plane (ComponentManagerService), a GitOps Bridge Controller for reconciling Kubernetes custom resources, a static linter for manifest validation, and Spanner-backed persistence. While the architecture is comprehensive, several critical issues must be addressed. Most notably, a Spanner parsing bug using a non-existent 'spanner.NullRow' type will cause immediate failures, and the deactivation flow is currently incomplete as 'CompleteDeactivation' is never invoked. Additionally, the reconciler exhibits destructive behavior by deleting workloads on transient registration failures, and the Spanner store suffers from a performance bottleneck due to full-table scans when querying by component class. Aligning snapshot state checks, handling ignored errors during deletion, and making the network prober more resilient to clustered environments are also highly recommended.
…resilience, and Spanner class index
Summary
Important
DRAFT / REFERENCE PR ONLY — DO NOT MERGE
This Pull Request contains the complete end-to-end prototype implementation of the XMC Components System based on the AI Components Design Document.
It is published as an immutable architectural reference for design doc reviewers, subagent evaluation, and subsequent task splitting into production PRs.
Motivation & Context
The XMC platform requires a unified mechanism to manage first-party (1P) and third-party (3P) auxiliary services (such as metrics ingestion pipelines, Kafka/Redpanda event streams, and ClickHouse analytics clusters).
This prototype validates Option C (Hybrid Cloud Spanner Source of Truth with GKE Informers), establishing:
deploy/profiles/stable-2026a/).xmc-linter).Component.xmc.dev) with CEL schema invariants and finalizer protection.xmc-bridge).ComponentManagerService(CMS) and Cloud Spanner.Step 1 Scope & ESS Co-location (Design Doc Alignment)
In the design document,
ComponentManagerService(CMS) is introduced as a logical control plane service. For Step 1 / Prototype, we explicitly decided to co-locate CMS inside the Experiment State Server (ESS) binary (cmd/experiment_state_server/):1. What Lives Inside ESS in Step 1:
componentspb.RegisterComponentManagerServiceServer).Components,ComponentDependencies, andExperimentComponentSnapshots.ExperimentComponentSnapshotsinterleaved inExperiments(BitReversedXid)) in a single in-process transaction without distributed 2PC or cross-service latency.2. What Lives Outside ESS in Step 1:
xmc-bridgeController: Runs as an independent in-cluster Kubernetes operator/daemon in GKE watchingComponent.xmc.devCRDs, reconciling 1P child workloads, running health probes, and syncing state with CMS via gRPC.xmc-linterCLI: Runs in pre-submit CI to statically validate component YAMLs, DAG cycles, and semver compatibility matrices.3. Rationale & Evolution Path:
ComponentManagerServiceproto andstore.Storeinterface, making extraction into an independent microservice trivial when independent scaling is required in the future.Expected Production GitOps Workflow
In production, the end-to-end user and infrastructure flow operates as follows:
flowchart TD subgraph GitOps_Repo["1. GitOps Repository (Admin / Platform Owner)"] GitCommit["PR / Commit to Git\n(deploy/profiles/stable-2026a/)"] CI["Pre-flight CI Validation\n(xmc-linter --manifests-dir ... --rules-config ...)"] GitCommit --> CI end subgraph CD_Sync["2. Continuous Delivery Pipeline"] SyncEngine["Config Sync / ArgoCD / Helm Controller"] CI -->|Merge to Main| SyncEngine end subgraph GKE_Cluster["3. GKE Control Plane & Workloads"] CRD["Component CRs (xmc.dev/v1alpha1)\nState: spec.enabled, parameters"] Bridge["XMC Bridge Controller (xmc-bridge)\nReconciler & Health Prober"] Workloads["1P Workloads (Deployments / Services)\n3P Workloads (Redpanda / ClickHouse)"] SyncEngine -->|Applies CRs| CRD CRD <-->|Watches / Updates Status| Bridge Bridge -->|Provisions / Probes| Workloads end subgraph Spanner_SOT["4. Cloud Spanner SOT & CMS"] CMS["ComponentManagerService (CMS)\n(experiment_state_server)"] SpannerDB[("Cloud Spanner Database\n- Components\n- ComponentDependencies\n- ExperimentComponentSnapshots")] Bridge <-->|SetComponentIntent / UpdateObservedGen| CMS CMS <-->|ReadWriteTransactions| SpannerDB end subgraph User_Flow["5. User Experiment Lifecycle"] UserCLI["User / SDK: xm.launch_experiment()"] LaunchT0["Experiment Admission ($T_0$)\nCMS.ResolveComponentSnapshot()"] JobSet["WorkUnit Execution / Pods\n(Injects resolved endpoints)"] UserCLI --> LaunchT0 LaunchT0 <-->|Atomic 2PL Interleaved Write| SpannerDB LaunchT0 --> JobSet JobSet -.->|Sends Metrics / Events| Workloads endStep-by-Step Operator & User Flow:
deploy/profiles/<profile>/components/<component>/values.yaml.xmc-linter.rules.yaml.Component.xmc.devCustom Resources to the GKE cluster.xmc-bridgedetects CR changes via dynamic informers.SetComponentIntent).DeploymentandServiceworkloads, attachingmetav1.OwnerReferencefor garbage collection.Ready,Degraded, andDeactivationBlockedconditions.CMS.ResolveComponentSnapshot(xid, requestedClasses).COMPONENT_STATE_ENABLEDstate.Experiments(BitReversedXid).Architectural Overview & Key Changes
1. Protobuf API & CMS gRPC Service
api/protos/components/v1alpha1/components.protoComponentManagerServicewith RPCs:SetComponentIntent: Synchronizes GitOps desired generation with Cloud Spanner.UpdateObservedGeneration: Updates in-cluster observed generation, access status, and condition state.PrepareDeactivation: Evaluates dependency graph safety, cascades, and drain timeouts.CompleteDeactivation: Commits state transition fromDISABLINGtoDISABLED.ExecuteDeletion: Soft-deletes catalog entry (State = DELETED), freeing uniqueActiveName.ResolveComponentSnapshot: Locks immutable component parameters at experiment launch.ValidateComponentRequirement: Validates pre-conditions for experiment submission.GetPlatformHealth: Telemetry and CDC replication lag metrics.2. Cloud Spanner Schema & Migration
010_components_system.up.sql010_components_system.down.sqlpkg/components/store/spanner_store.goComponents: Primary catalog table with random 64-bit PKs and null-filtered unique indexIdx_Components_ActiveName.ComponentDependencies: Interleaved dependency graph table with foreign key constraints.ExperimentComponentSnapshots: Interleaved inExperimentsonBitReversedXidfor atomic single-split commits.3. GitOps Bridge Controller (
xmc-bridge)cmd/xmc-bridge/main.gopkg/components/bridge/reconciler.gopkg/components/bridge/workload_manager.gopkg/components/bridge/prober.goComponent.xmc.devCRDs.DeploymentandServiceworkloads withOwnerReferences.finalizers.xmc.dev/deletion-protection).4. Static Linter CLI (
xmc-linter)cmd/xmc-linter/main.gopkg/components/linter/linter.go>=,>,<=,<,^).5. Profile Components Migrated
deploy/profiles/stable-2026a/components/clickhouse/(3P Wrapper)deploy/profiles/stable-2026a/components/redpanda/(3P Wrapper)deploy/profiles/stable-2026a/components/ingestion-gateway/(1P Native Workload)Verification & Testing
1. Hermetic Bazel Test Suites
All 5 component unit and integration test packages pass hermetically:
bazel test //api/protos/components/... //pkg/components/store:... //pkg/components/cms:... //pkg/components/bridge:... //pkg/components/linter:... //pkg/components/summary:...store_test.go): Memory and Spanner store transactions, soft-deletions, dependency cascades, and snapshot resolution.server_test.go): gRPC status codes, precondition validations, and error masking.reconciler_test.go): Status condition transitions, health probe simulations, and workload lifecycle.linter_test.go): Manifest validation, cyclic dependency detection, and semver matrix checking.info_provider_test.go): Telemetry aggregation.2. Live GKE Cluster & Cloud Spanner End-to-End Verification
Deployed images
xmc-bridge:latestand...:latestto GKE cluster and verified:redpandawithcascade: falseblocked by CMS (DeactivationBlocked=True, active dependentingestion-gatewaydetected).redpandawithcascade: truesuccessfully transitionedredpandatoDISABLEDand triggered automatic teardown ofingestion-gateway's 1PDeploymentandService.clickhouseingestion-gateway. Caught immediately byxmc-linterduring pre-flight.kubectl delete component clickhousewhiledeletionProtection: true. Blocked by finalizer withDeletionBlocked=True.deletionProtection: false; component cleanly soft-deleted in Spanner (State = DELETED) and removed from GKE.parameters.portoningestion-gatewayfrom8080to8088. Controller performed rolling update on child Deployment and Service without downtime or recreation conflicts.spec.wrapperRefis rejected at API server admission time.Suggested Task Breakdown for Production PRs
To land this design cleanly in production, it is recommended to split this prototype into at least 6 scoped PRs (as individual changes might get too big):
api/protos/components/v1alpha1/components.protopkg/components/store/store.gointernal/storage/spanner/migrations/ess/010_components_system.*.sqlpkg/components/store/spanner_store.go,memory_store.go,store_test.gopkg/components/cms/server.go,server_test.gocmd/experiment_state_server/main.gopkg/components/linter/,cmd/xmc-linter/,rules.yamldeploy/crd/component_crd.yaml,cmd/xmc-bridge/,pkg/components/bridge/deploy/profiles/stable-2026a/(ClickHouse, Redpanda, Ingestion Gateway)Fixes bug N/A (Prototype & Architectural Reference)
TAG=agy
CONV=59b0a148-0a1a-4bd2-a98a-6f0c5c7e8e1a