Skip to content

fix(bridge): gate component actuation on dependency readiness - #991

Merged
VLukiantsev merged 3 commits into
fix/bridge-1p-health-and-hydrationfrom
feat/bridge-dependency-gating
Sep 28, 2026
Merged

VLukiantsev merged 3 commits into
fix/bridge-1p-health-and-hydrationfrom
feat/bridge-dependency-gating

Conversation

@VLukiantsev

@VLukiantsev VLukiantsev commented Sep 25, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

xmc-bridge no longer actuates a component until the components in its spec.dependencies are Ready. The component is still registered in CMS right away, so its dependency edges protect its dependencies from deactivation.

Stack 3/4: CMS dependencies (0) → 1P health (B) → this PR → dependency-gated deactivation (C2).

Motivation & Context

The bridge installed 1P workloads (e.g. ingestion-gateway) even when their dependencies (redpanda) were missing, disabled or unhealthy. On Spanner, an unregistered dependency also caused IntentApplicationFailed noise.

An earlier version of this change gated before registering the intent. A live test then showed that CMS never learned the dependency edge, so disabling redpanda was not blocked. This version therefore separates registration from actuation.

Key Changes

  • DependencySnapshot, built once per pass from the Component list the controller already fetches. A dependency is ready when all of these hold:

    • it exists and has spec.enabled set;
    • it is not being deleted;
    • Ready=True at the current generation;
    • it is not Degraded.

    Multi-node cycles are detected and reported as DependencyCycle (Degraded=True).

  • Register intent first, then gate actuation. A Spanner FailedPrecondition "dependency not found" counts as waiting (DependenciesNotReady, Degraded=False, no error log).

  • Not-yet-actuated components are blocked: PENDING_RECONCILIATION, Ready=False/DependenciesNotReady, with a message such as waiting on dependencies: redpanda (disabled).

  • Already-running components keep their health-derived Ready and get a DependenciesReady=False condition. Helm upgrades are held and the running workload is left in place. A held upgrade reports HelmUpgradePending.

  • Wiring: ReconcileWithDependencies plus the controller wiring. Reconcile keeps its signature (nil resolver means no gating).

  • Docs: READMEs cover the gating rules, the new conditions and reasons, and that N-deep chains settle over up to N passes.

Assumptions

  • A running dependent is left running when a dependency becomes unready; only upgrades are held. — User decision. — A dependent may run against a degraded dependency until it recovers.
  • The gate applies to 1P and 3P components. — Dependency semantics don't depend on type, and a 3P wrapper shouldn't report Ready before its dependencies do.
  • Readiness uses the dependency's Ready condition. — For 1P this means "Helm applied the manifests", not pod readiness (see PR B).

Verification & Testing

  • Commands Executed: bazel test //pkg/components/... //cmd/xmc_bridge/... (9/9 pass), ./.githooks/pre-commit (pass). Size vs parent: 294 prod / 890 total.
  • New tests:
    • snapshot readiness and cycles;
    • activation gating: missing, disabled, not ready, stale generation, degraded, all ready;
    • a gated dependent still sends SetComponentIntent with its dependencies;
    • an unregistered-dependency rejection gates without error;
    • running components keep Ready;
    • multi-pass blip → held upgrade → recovery;
    • controller wiring.
  • Live (GKE ...), only toggling spec.enabled:
    • enabling ingestion-gateway with redpanda disabled gave PENDING/DependenciesNotReady (waiting on dependencies: redpanda (disabled)), with no Helm release created;
    • re-enabling redpanda led the bridge to install xmc-cmp-ingestion-gateway on its own, reaching ENABLED/Ready=True.

Related Issues / Bugs

Found while working on bug 548435582.

TAG=agy
CONV=baca6192-74d3-4579-9f73-0a3263d9a2a7

@VLukiantsev
VLukiantsev added this pull request to stack #993 September 25, 2026 12:12

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces dependency activation gating for Component custom resources in the xmc-bridge controller. It adds a new DependencySnapshot mechanism to evaluate dependency readiness and detect multi-node dependency cycles. The Reconciler now gates actuation of enabled components on their dependencies being ready, holding upgrades or initial deployments when dependencies are missing, disabled, or unready, while keeping existing workloads running during temporary dependency blips. Comprehensive unit tests and documentation updates have been added to support this feature. No review comments were provided, so there is no feedback to address.

@VLukiantsev

Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements dependency activation gating for Component CRs in the GitOps bridge. It introduces a DependencySnapshot to evaluate the readiness of dependencies and detect multi-node dependency cycles. Actuation of enabled components is gated on dependency readiness: not-yet-actuated components are held in a pending state, while already-running components hold pending Helm upgrades without tearing down existing workloads. The feedback suggests optimizing performance by avoiding redundant parsing of unstructured specs in NewDependencySnapshot, using the idiomatic status.FromError for gRPC status extraction, and refining the status condition message to avoid mentioning Helm upgrades for 3P wrapper components.

Comment thread pkg/components/bridge/reconciler.go Resolved
Comment thread pkg/components/bridge/reconciler.go Resolved
@VLukiantsev

Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces dependency activation gating in the GitOps bridge (xmc_bridge). It implements a DependencySnapshot mechanism to evaluate the readiness of declared component dependencies and detect dependency cycles before actuating workloads. Enabled components with unready dependencies are held in a pending state (COMPONENT_STATE_PENDING_RECONCILIATION with reason DependenciesNotReady), while already-actuated components hold pending Helm upgrades without tearing down running workloads. The changes include the new DependencyReadiness interface, cycle detection integration, updated reconciliation logic, updated documentation, and comprehensive unit tests. There are no review comments to evaluate, so I have no feedback to provide.

@VLukiantsev
VLukiantsev force-pushed the feat/bridge-dependency-gating branch from 19df223 to 32bc3c3 Compare September 25, 2026 12:39
@VLukiantsev
VLukiantsev force-pushed the feat/bridge-dependency-gating branch from 32bc3c3 to 3178da5 Compare September 25, 2026 13:19
@VLukiantsev
VLukiantsev marked this pull request as ready for review September 25, 2026 14:47
@VLukiantsev
VLukiantsev removed this pull request from stack #993 September 25, 2026 14:49
@VLukiantsev
VLukiantsev added this pull request to stack #995 September 25, 2026 14:49
@VLukiantsev
VLukiantsev force-pushed the feat/bridge-dependency-gating branch 2 times, most recently from ce5373d to ed981c2 Compare September 28, 2026 08:26
@VLukiantsev VLukiantsev added the priority::2-high top of mind. should not wait. label Sep 28, 2026
andrzejfudro
andrzejfudro previously approved these changes Sep 28, 2026
github-actions[bot]
github-actions Bot previously approved these changes Sep 28, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Incremental PR Approval

Clean rebase detected with zero semantic patch lines changed.

Preserving prior approval by @andrzejfudro on commit ed981c2.

The xmc-bridge now orders activation by spec.dependencies:

- cmd/xmc_bridge builds a DependencySnapshot once per pass and calls
  ReconcileWithDependencies. A dependency is ready only if its CR exists,
  is enabled, is not deleting, reports Ready=True at its current
  generation, and is not Degraded.
- Enabled components always register intent (with dependency edges)
  with CMS first, then gate actuation. A CMS FailedPrecondition for an
  unregistered dependency is treated as waiting, not as an error.
- Not-yet-actuated components wait in PENDING_RECONCILIATION with
  Ready=False reason DependenciesNotReady; no workload is created.
- Actuated components are never torn down by a dependency blip. They
  report DependenciesReady=False and Helm upgrades are held. A held
  upgrade to a new generation reports Ready=False/Degraded=False reason
  HelmUpgradePending instead of claiming readiness at that generation.
- Components on a dependency cycle report DependencyCycle.

BUG=b/548435582
TAG=agy
…y message

- NewDependencySnapshot parses each Component spec once and passes the
  enabled flag to componentUnreadyReason.
- DependenciesReady=False only mentions held Helm upgrades for 1P
  components; 3P wrappers have no Helm release.
- Document why dependencyNotRegisteredMessage uses errors.As rather than
  status.FromError (wrapped errors lose the original status message).

Comment link: #991 (comment)
Comment link: #991 (comment)
Comment link: #991 (comment)
Comment link: #991 (comment)

BUG=b/548435582
TAG=agy
CONV=baca6192-74d3-4579-9f73-0a3263d9a2a7
The status write replaces all conditions, so an error pass dropped
DependenciesReady and isActuated could treat an actuated but unhealthy
3P wrapper as never actuated on the next pass, re-gating it. Carry the
existing condition over registerIntent/actuation errors, and document
that the dependency snapshot is namespace-scoped.

BUG=b/548435582
TAG=agy
CONV=baca6192-74d3-4579-9f73-0a3263d9a2a7

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Incremental PR Approval

PR author is a code owner and changes since last approval (4 lines) are within the 10-line grace delta.

Preserving prior approval by @andrzejfudro on commit ed981c2.

@VLukiantsev
VLukiantsev added this pull request to the merge queue Sep 28, 2026
Merged via the queue into main with commit 8f7c6e9 Sep 28, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority::2-high top of mind. should not wait.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants