Skip to content

feat(components): prototype XMC components system with GitOps, CMS, and Spanner SOT - #456

Closed
VLukiantsev wants to merge 2 commits into
mainfrom
feat/implement-xmc-components-prototype
Closed

VLukiantsev wants to merge 2 commits into
mainfrom
feat/implement-xmc-components-prototype

Conversation

@VLukiantsev

@VLukiantsev VLukiantsev commented Aug 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Important

DRAFT / REFERENCE PR ONLY — DO NOT MERGE
This Pull Request contains the complete end-to-end prototype implementation of the XMC Components System based on the AI Components Design Document.
It is published as an immutable architectural reference for design doc reviewers, subagent evaluation, and subsequent task splitting into production PRs.


Motivation & Context

The XMC platform requires a unified mechanism to manage first-party (1P) and third-party (3P) auxiliary services (such as metrics ingestion pipelines, Kafka/Redpanda event streams, and ClickHouse analytics clusters).
This prototype validates Option C (Hybrid Cloud Spanner Source of Truth with GKE Informers), establishing:

  1. Declarative, versioned GitOps profile configurations (deploy/profiles/stable-2026a/).
  2. Pre-flight static validation with DAG cycle detection and semver compatibility checking (xmc-linter).
  3. Kubernetes Custom Resource Definitions (Component.xmc.dev) with CEL schema invariants and finalizer protection.
  4. Bidirectional state reconciliation and endpoint probing via the GitOps Bridge Controller (xmc-bridge).
  5. Immutable, bit-reversed experiment snapshot locking at launch time $T_0$ via ComponentManagerService (CMS) and Cloud Spanner.

Step 1 Scope & ESS Co-location (Design Doc Alignment)

In the design document, ComponentManagerService (CMS) is introduced as a logical control plane service. For Step 1 / Prototype, we explicitly decided to co-locate CMS inside the Experiment State Server (ESS) binary (cmd/experiment_state_server/):

1. What Lives Inside ESS in Step 1:

  • CMS gRPC Server: Registered directly on ESS's existing gRPC server port (componentspb.RegisterComponentManagerServiceServer).
  • Shared Cloud Spanner SOT: Reuses ESS's Spanner connection pool to manage Components, ComponentDependencies, and ExperimentComponentSnapshots.
  • In-Process $T_0$ Snapshot Resolution: When experiments are admitted/launched, ESS resolves component dependencies and commits immutable snapshot rows (ExperimentComponentSnapshots interleaved in Experiments(BitReversedXid)) in a single in-process transaction without distributed 2PC or cross-service latency.

2. What Lives Outside ESS in Step 1:

  • xmc-bridge Controller: Runs as an independent in-cluster Kubernetes operator/daemon in GKE watching Component.xmc.dev CRDs, reconciling 1P child workloads, running health probes, and syncing state with CMS via gRPC.
  • xmc-linter CLI: Runs in pre-submit CI to statically validate component YAMLs, DAG cycles, and semver compatibility matrices.

3. Rationale & Evolution Path:

  • Zero Distributed Transaction Complexity: Eliminates 2-Phase Commit overhead between a separate CMS service and ESS at experiment launch time ($T_0$).
  • Reduced Operational Footprint: No new deployments, load balancers, or separate Spanner IAM roles required for Step 1.
  • Seamless Step 2+ Extraction Path: CMS logic and storage access are strictly encapsulated behind the ComponentManagerService proto and store.Store interface, making extraction into an independent microservice trivial when independent scaling is required in the future.

Expected Production GitOps Workflow

In production, the end-to-end user and infrastructure flow operates as follows:

flowchart TD
    subgraph GitOps_Repo["1. GitOps Repository (Admin / Platform Owner)"]
        GitCommit["PR / Commit to Git\n(deploy/profiles/stable-2026a/)"]
        CI["Pre-flight CI Validation\n(xmc-linter --manifests-dir ... --rules-config ...)"]
        GitCommit --> CI
    end

    subgraph CD_Sync["2. Continuous Delivery Pipeline"]
        SyncEngine["Config Sync / ArgoCD / Helm Controller"]
        CI -->|Merge to Main| SyncEngine
    end

    subgraph GKE_Cluster["3. GKE Control Plane & Workloads"]
        CRD["Component CRs (xmc.dev/v1alpha1)\nState: spec.enabled, parameters"]
        Bridge["XMC Bridge Controller (xmc-bridge)\nReconciler & Health Prober"]
        Workloads["1P Workloads (Deployments / Services)\n3P Workloads (Redpanda / ClickHouse)"]
        
        SyncEngine -->|Applies CRs| CRD
        CRD <-->|Watches / Updates Status| Bridge
        Bridge -->|Provisions / Probes| Workloads
    end

    subgraph Spanner_SOT["4. Cloud Spanner SOT & CMS"]
        CMS["ComponentManagerService (CMS)\n(experiment_state_server)"]
        SpannerDB[("Cloud Spanner Database\n- Components\n- ComponentDependencies\n- ExperimentComponentSnapshots")]
        
        Bridge <-->|SetComponentIntent / UpdateObservedGen| CMS
        CMS <-->|ReadWriteTransactions| SpannerDB
    end

    subgraph User_Flow["5. User Experiment Lifecycle"]
        UserCLI["User / SDK: xm.launch_experiment()"]
        LaunchT0["Experiment Admission ($T_0$)\nCMS.ResolveComponentSnapshot()"]
        JobSet["WorkUnit Execution / Pods\n(Injects resolved endpoints)"]
        
        UserCLI --> LaunchT0
        LaunchT0 <-->|Atomic 2PL Interleaved Write| SpannerDB
        LaunchT0 --> JobSet
        JobSet -.->|Sends Metrics / Events| Workloads
    end
Loading

Step-by-Step Operator & User Flow:

  1. Operator Modifies Component Intent:
    • The platform admin enables/disables a component or tunes parameters in deploy/profiles/<profile>/components/<component>/values.yaml.
  2. Pre-flight CI Validation Gate:
    • Pre-submit CI executes xmc-linter.
    • Pass 1: Validates manifest schema, labels, and deletion protection flags.
    • Pass 2: Builds topological dependency graph and fails immediately if circular dependencies exist ($A \to B \to A$).
    • Pass 3: Enforces semantic version compatibility matrices against rules.yaml.
  3. Automated CD Delivery:
    • On merge, the GitOps engine (e.g. Config Sync, ArgoCD, or Helm) applies rendered Component.xmc.dev Custom Resources to the GKE cluster.
  4. Bridge Controller In-Cluster Reconciliation:
    • xmc-bridge detects CR changes via dynamic informers.
    • Submits declared intent to CMS (SetComponentIntent).
    • If 1P component: Dynamically reconciles child Deployment and Service workloads, attaching metav1.OwnerReference for garbage collection.
    • If 3P component: Observes foreign CR status or performs network TCP/HTTP health probes.
    • Preserves condition timestamps (KEP-1623) and updates Ready, Degraded, and DeactivationBlocked conditions.
  5. Experiment Launch Time ($T_0$) Snapshot Resolution:
    • When a user submits an experiment, ESS calls CMS.ResolveComponentSnapshot(xid, requestedClasses).
    • CMS validates that required components are in COMPONENT_STATE_ENABLED state.
    • Resolves transitive dependency graph outside 2PL lock contention, then writes atomic, immutable snapshot rows interleaved in Experiments(BitReversedXid).
    • WorkUnits receive exact runtime connection parameters that remain immutable even if the platform catalog changes later.

Architectural Overview & Key Changes

1. Protobuf API & CMS gRPC Service

  • File: api/protos/components/v1alpha1/components.proto
  • Defined ComponentManagerService with RPCs:
    • SetComponentIntent: Synchronizes GitOps desired generation with Cloud Spanner.
    • UpdateObservedGeneration: Updates in-cluster observed generation, access status, and condition state.
    • PrepareDeactivation: Evaluates dependency graph safety, cascades, and drain timeouts.
    • CompleteDeactivation: Commits state transition from DISABLING to DISABLED.
    • ExecuteDeletion: Soft-deletes catalog entry (State = DELETED), freeing unique ActiveName.
    • ResolveComponentSnapshot: Locks immutable component parameters at experiment launch.
    • ValidateComponentRequirement: Validates pre-conditions for experiment submission.
    • GetPlatformHealth: Telemetry and CDC replication lag metrics.

2. Cloud Spanner Schema & Migration

3. GitOps Bridge Controller (xmc-bridge)

4. Static Linter CLI (xmc-linter)

5. Profile Components Migrated

  • Files:
    • deploy/profiles/stable-2026a/components/clickhouse/ (3P Wrapper)
    • deploy/profiles/stable-2026a/components/redpanda/ (3P Wrapper)
    • deploy/profiles/stable-2026a/components/ingestion-gateway/ (1P Native Workload)

Verification & Testing

1. Hermetic Bazel Test Suites

All 5 component unit and integration test packages pass hermetically:

bazel test //api/protos/components/... //pkg/components/store:... //pkg/components/cms:... //pkg/components/bridge:... //pkg/components/linter:... //pkg/components/summary:...
  • Store Tests (store_test.go): Memory and Spanner store transactions, soft-deletions, dependency cascades, and snapshot resolution.
  • CMS Tests (server_test.go): gRPC status codes, precondition validations, and error masking.
  • Bridge Tests (reconciler_test.go): Status condition transitions, health probe simulations, and workload lifecycle.
  • Linter Tests (linter_test.go): Manifest validation, cyclic dependency detection, and semver matrix checking.
  • Summary Tests (info_provider_test.go): Telemetry aggregation.

2. Live GKE Cluster & Cloud Spanner End-to-End Verification

Deployed images xmc-bridge:latest and ...:latest to GKE cluster and verified:

  1. Cascade vs. Non-Cascade Deactivation:
    • Disabling redpanda with cascade: false blocked by CMS (DeactivationBlocked=True, active dependent ingestion-gateway detected).
    • Disabling redpanda with cascade: true successfully transitioned redpanda to DISABLED and triggered automatic teardown of ingestion-gateway's 1P Deployment and Service.
  2. Circular Dependency Detection:
    • Tested intentional cyclic dependency clickhouse $\leftrightarrow$ ingestion-gateway. Caught immediately by xmc-linter during pre-flight.
  3. Deletion Protection & Finalizers:
    • Attempted kubectl delete component clickhouse while deletionProtection: true. Blocked by finalizer with DeletionBlocked=True.
    • Patched deletionProtection: false; component cleanly soft-deleted in Spanner (State = DELETED) and removed from GKE.
  4. Live Workload Parameter Mutation:
    • Mutated parameters.port on ingestion-gateway from 8080 to 8088. Controller performed rolling update on child Deployment and Service without downtime or recreation conflicts.
  5. CRD CEL Admission Invariants:
    • Verified that creating 3P wrappers without spec.wrapperRef is rejected at API server admission time.

Suggested Task Breakdown for Production PRs

To land this design cleanly in production, it is recommended to split this prototype into at least 6 scoped PRs (as individual changes might get too big):

  1. PR 1: Protobuf API & Core Interfaces
    • api/protos/components/v1alpha1/components.proto
    • pkg/components/store/store.go
  2. PR 2: Cloud Spanner Migrations & Store Implementation
    • internal/storage/spanner/migrations/ess/010_components_system.*.sql
    • pkg/components/store/spanner_store.go, memory_store.go, store_test.go
  3. PR 3: ComponentManagerService (CMS) in Experiment State Server
    • pkg/components/cms/server.go, server_test.go
    • Wiring in cmd/experiment_state_server/main.go
  4. PR 4: Static Linter CLI & Pre-flight CI Rules
    • pkg/components/linter/, cmd/xmc-linter/, rules.yaml
  5. PR 5: Custom Resource Definition & GitOps Bridge Controller
    • deploy/crd/component_crd.yaml, cmd/xmc-bridge/, pkg/components/bridge/
  6. PR 6: Platform Release Profiles & Component Charts
    • deploy/profiles/stable-2026a/ (ClickHouse, Redpanda, Ingestion Gateway)

Fixes bug N/A (Prototype & Architectural Reference)

TAG=agy
CONV=59b0a148-0a1a-4bd2-a98a-6f0c5c7e8e1a

…nd Spanner SOT

Implements the foundational prototype for the XMC Components System based on the AI Components Design Document (Option C: Hybrid Spanner SOT with GKE Informers).

Key Capabilities:
- ComponentManagerService (CMS) Protobuf gRPC definitions and server implementation.
- Cloud Spanner migration (010 up/down) with interleaved dependencies and bit-reversed experiment snapshot locking (T0).
- GitOps Bridge Controller (cmd/xmc-bridge) for reconciling Component CRDs, dynamic health probing, and 1P workload management.
- 3-pass static manifest linter (xmc-linter) for syntax, DAG cycle detection, and semver compatibility matrix enforcement.
- Initial component migrations for ClickHouse (3P wrapper), Redpanda (3P wrapper), and Ingestion Gateway (1P native).

TAG=agy
CONV=59b0a148-0a1a-4bd2-a98a-6f0c5c7e8e1a

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the XMC Components System, which includes a gRPC control plane (ComponentManagerService), a GitOps Bridge Controller for reconciling Kubernetes custom resources, a static linter for manifest validation, and Spanner-backed persistence. While the architecture is comprehensive, several critical issues must be addressed. Most notably, a Spanner parsing bug using a non-existent 'spanner.NullRow' type will cause immediate failures, and the deactivation flow is currently incomplete as 'CompleteDeactivation' is never invoked. Additionally, the reconciler exhibits destructive behavior by deleting workloads on transient registration failures, and the Spanner store suffers from a performance bottleneck due to full-table scans when querying by component class. Aligning snapshot state checks, handling ignored errors during deletion, and making the network prober more resilient to clustered environments are also highly recommended.

Comment thread pkg/components/store/spanner_store.go Resolved
Comment thread pkg/components/bridge/reconciler.go Resolved
Comment thread pkg/components/bridge/reconciler.go Resolved
Comment thread pkg/components/store/spanner_store.go Resolved
Comment thread pkg/components/bridge/reconciler.go Resolved
Comment thread pkg/components/store/spanner_store.go Resolved
Comment thread pkg/components/bridge/reconciler.go Resolved
Comment thread pkg/components/bridge/prober.go Resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant