Repository navigation
fix(bridge): gate component actuation on dependency readiness - #991
VLukiantsev merged 3 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces dependency activation gating for Component custom resources in the xmc-bridge controller. It adds a new DependencySnapshot mechanism to evaluate dependency readiness and detect multi-node dependency cycles. The Reconciler now gates actuation of enabled components on their dependencies being ready, holding upgrades or initial deployments when dependencies are missing, disabled, or unready, while keeping existing workloads running during temporary dependency blips. Comprehensive unit tests and documentation updates have been added to support this feature. No review comments were provided, so there is no feedback to address.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request implements dependency activation gating for Component CRs in the GitOps bridge. It introduces a DependencySnapshot to evaluate the readiness of dependencies and detect multi-node dependency cycles. Actuation of enabled components is gated on dependency readiness: not-yet-actuated components are held in a pending state, while already-running components hold pending Helm upgrades without tearing down existing workloads. The feedback suggests optimizing performance by avoiding redundant parsing of unstructured specs in NewDependencySnapshot, using the idiomatic status.FromError for gRPC status extraction, and refining the status condition message to avoid mentioning Helm upgrades for 3P wrapper components.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces dependency activation gating in the GitOps bridge (xmc_bridge). It implements a DependencySnapshot mechanism to evaluate the readiness of declared component dependencies and detect dependency cycles before actuating workloads. Enabled components with unready dependencies are held in a pending state (COMPONENT_STATE_PENDING_RECONCILIATION with reason DependenciesNotReady), while already-actuated components hold pending Helm upgrades without tearing down running workloads. The changes include the new DependencyReadiness interface, cycle detection integration, updated reconciliation logic, updated documentation, and comprehensive unit tests. There are no review comments to evaluate, so I have no feedback to provide.
19df223 to
32bc3c3
Compare
32bc3c3 to
3178da5
Compare
ce5373d to
ed981c2
Compare
ed981c2 to
4572d8b
Compare
There was a problem hiding this comment.
🤖 Incremental PR Approval
Clean rebase detected with zero semantic patch lines changed.
Preserving prior approval by @andrzejfudro on commit ed981c2.
The xmc-bridge now orders activation by spec.dependencies: - cmd/xmc_bridge builds a DependencySnapshot once per pass and calls ReconcileWithDependencies. A dependency is ready only if its CR exists, is enabled, is not deleting, reports Ready=True at its current generation, and is not Degraded. - Enabled components always register intent (with dependency edges) with CMS first, then gate actuation. A CMS FailedPrecondition for an unregistered dependency is treated as waiting, not as an error. - Not-yet-actuated components wait in PENDING_RECONCILIATION with Ready=False reason DependenciesNotReady; no workload is created. - Actuated components are never torn down by a dependency blip. They report DependenciesReady=False and Helm upgrades are held. A held upgrade to a new generation reports Ready=False/Degraded=False reason HelmUpgradePending instead of claiming readiness at that generation. - Components on a dependency cycle report DependencyCycle. BUG=b/548435582 TAG=agy
…y message - NewDependencySnapshot parses each Component spec once and passes the enabled flag to componentUnreadyReason. - DependenciesReady=False only mentions held Helm upgrades for 1P components; 3P wrappers have no Helm release. - Document why dependencyNotRegisteredMessage uses errors.As rather than status.FromError (wrapped errors lose the original status message). Comment link: #991 (comment) Comment link: #991 (comment) Comment link: #991 (comment) Comment link: #991 (comment) BUG=b/548435582 TAG=agy CONV=baca6192-74d3-4579-9f73-0a3263d9a2a7
The status write replaces all conditions, so an error pass dropped DependenciesReady and isActuated could treat an actuated but unhealthy 3P wrapper as never actuated on the next pass, re-gating it. Carry the existing condition over registerIntent/actuation errors, and document that the dependency snapshot is namespace-scoped. BUG=b/548435582 TAG=agy CONV=baca6192-74d3-4579-9f73-0a3263d9a2a7
4572d8b to
ee2f8e4
Compare
There was a problem hiding this comment.
🤖 Incremental PR Approval
PR author is a code owner and changes since last approval (4 lines) are within the 10-line grace delta.
Preserving prior approval by @andrzejfudro on commit ed981c2.
Summary
xmc-bridge no longer actuates a component until the components in its
spec.dependenciesare Ready. The component is still registered in CMS right away, so its dependency edges protect its dependencies from deactivation.Stack 3/4: CMS dependencies (0) → 1P health (B) → this PR → dependency-gated deactivation (C2).
Motivation & Context
The bridge installed 1P workloads (e.g.
ingestion-gateway) even when their dependencies (redpanda) were missing, disabled or unhealthy. On Spanner, an unregistered dependency also causedIntentApplicationFailednoise.An earlier version of this change gated before registering the intent. A live test then showed that CMS never learned the dependency edge, so disabling
redpandawas not blocked. This version therefore separates registration from actuation.Key Changes
DependencySnapshot, built once per pass from the Component list the controller already fetches. A dependency is ready when all of these hold:spec.enabledset;Ready=Trueat the current generation;Degraded.Multi-node cycles are detected and reported as
DependencyCycle(Degraded=True).Register intent first, then gate actuation. A Spanner
FailedPrecondition "dependency not found"counts as waiting (DependenciesNotReady, Degraded=False, no error log).Not-yet-actuated components are blocked:
PENDING_RECONCILIATION,Ready=False/DependenciesNotReady, with a message such aswaiting on dependencies: redpanda (disabled).Already-running components keep their health-derived
Readyand get aDependenciesReady=Falsecondition. Helm upgrades are held and the running workload is left in place. A held upgrade reportsHelmUpgradePending.Wiring:
ReconcileWithDependenciesplus the controller wiring.Reconcilekeeps its signature (nil resolver means no gating).Docs: READMEs cover the gating rules, the new conditions and reasons, and that N-deep chains settle over up to N passes.
Assumptions
Readycondition. — For 1P this means "Helm applied the manifests", not pod readiness (see PR B).Verification & Testing
bazel test //pkg/components/... //cmd/xmc_bridge/...(9/9 pass),./.githooks/pre-commit(pass). Size vs parent: 294 prod / 890 total.SetComponentIntentwith its dependencies;Ready;...), only togglingspec.enabled:ingestion-gatewaywithredpandadisabled gavePENDING/DependenciesNotReady(waiting on dependencies: redpanda (disabled)), with no Helm release created;redpandaled the bridge to installxmc-cmp-ingestion-gatewayon its own, reachingENABLED/Ready=True.Related Issues / Bugs
Found while working on bug 548435582.
TAG=agy
CONV=baca6192-74d3-4579-9f73-0a3263d9a2a7