Skip to content

feat(components): implement custom components operational health reporting - #792

Merged
VLukiantsev merged 6 commits into
mainfrom
feat/components-health-status-flow-b548436559
Sep 9, 2026
Merged

VLukiantsev merged 6 commits into
mainfrom
feat/components-health-status-flow-b548436559

Conversation

@VLukiantsev

@VLukiantsev VLukiantsev commented Sep 7, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Layer 1 of the Custom Components Admin Visibility stack (b/548436559). Introduces pkg/components/summary CustomComponentsInfoProvider library querying component state directly and exclusively from the Spanner Store in-memory, and integrates operational health evaluation into cmd/health_service GetHealthStatus.

Motivation & Context

The XMC Admin Panel requires visibility into installed Custom Components (1P native services and 3P Kubernetes CRD wrappers). To eliminate external RPC latency bottlenecks and single points of failure on the health check path, this change reads component state exclusively from the Spanner Store (store.Store.ListComponents).

It integrates this provider into cmd/health_service behind the --enable_components feature flag (health_enable_components env var), exposing custom component operational status in GetHealthStatus (ComponentType_COMPONENT_TYPE_CUSTOM_COMPONENT).

Key Changes

  • pkg/components/summary:
    • Defined CustomComponentsInfoProvider interface and StoreComponentsProvider backed by store.Store.
    • Implemented in-memory operational health evaluation (GetComponentsHealth), mapping component states (ENABLED, PENDING_RECONCILIATION, DISABLED, DISABLING) and reconciler readiness conditions to HealthStateHealthy, HealthStateUnhealthy, or HealthStateDisabled.
    • Implemented String() methods for ComponentType and HealthState enums and compile-time interface assertions.
  • cmd/health_service:
    • config.go: Added EnableComponents bool loaded via configutil.BoolEnv and -enable_components flag.
    • health_checker.go: Added ComponentsProvider to HealthCheckerParams; parallelized custom component health checks inside RunChecks alongside Spanner checks, converting results into ComponentType_COMPONENT_TYPE_CUSTOM_COMPONENT entries. Mapped disabled custom components to HEALTH_STATE_UNSPECIFIED.
    • server.go: Added dependency validation in newServer; updated calculateHealthStatus to only flag overall platform health as unhealthy on explicit HEALTH_STATE_UNHEALTHY component states.
    • main.go: Added linear helper initCustomComponentsProvider with graceful error recovery; ensured fresh timeout context for deferred deps.Close.
  • Unit & Concurrency Tests:
    • Added hermetic unit tests in pkg/components/summary/info_provider_test.go and cmd/health_service/health_checker_test.go covering all positive, error, nil-status, disabled component, and concurrency race detection scenarios.

Assumptions

Verification & Testing

  • Bazel Unit Tests:
    • bazel test //pkg/components/summary:all //cmd/health_service:all passed cleanly.
  • Concurrency & Race Detection:
    • bazel test --@io_bazel_rules_go//go/config:race --nocache_test_results //cmd/health_service:all //pkg/components/summary:all passed with 0 data races.
  • Diff Sizing:
    • Production code: 342 lines (< 400 threshold)
    • Test code: 427 lines (55.5% test ratio)
    • Total code: 769 lines (< 1,000 threshold)
  • Pre-Commit Linters:
    • Buildifier, gofmt, golangci-lint, Go test determinism checker, and addlicense passed cleanly (./.githooks/pre-commit).

Related Issues / Bugs

Part of b/548436559

TAG=agy
CONV=2181912f-1e25-4102-af4e-a8ae3e9b2195

@VLukiantsev VLukiantsev changed the title feat/components health status flow b548436559 feat(components): implement custom components operational health reporting Sep 7, 2026
@VLukiantsev

Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request integrates custom components discovery and health checking into the health service, introducing a new summary package to retrieve and evaluate custom component health states. The health checker is updated to run these checks in parallel with existing Spanner checks. The review feedback highlights two important issues: first, changing the overall health check logic to only trigger on HEALTH_STATE_UNHEALTHY risks masking core service failures that report as UNSPECIFIED, so UNSPECIFIED states should only be ignored for custom components. Second, if custom components are enabled, initialization failures for the Spanner client or store provider should result in a fatal error rather than silently disabling the provider.

Comment thread cmd/health_service/server.go Resolved
Comment thread cmd/health_service/main.go Resolved

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces custom components discovery and health checking to the health service, allowing it to query and report the health of custom 1P and 3P components alongside core services. The feedback focuses on improving robustness: first, by adding a dedicated 3-second timeout to the custom components health check to prevent slow providers from delaying the entire health check response; and second, by adding defensive nil checks in the health status calculation loop to avoid potential nil-pointer dereference panics.

Comment thread cmd/health_service/health_checker.go Resolved
Comment thread cmd/health_service/server.go Resolved
@VLukiantsev

Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request integrates custom components discovery and health checking into the health service. It introduces a new summary package containing StoreComponentsProvider to retrieve and evaluate custom component health statuses from a backing store. The health checker is updated to query these custom components in parallel with core Spanner checks, and the server is adjusted to prevent disabled custom components from degrading the overall health status. Feedback suggests adding a nil check for components in calculateHealthStatus to avoid potential nil pointer dereferences.

Comment thread cmd/health_service/server.go Resolved
@VLukiantsev
VLukiantsev force-pushed the feat/components-health-status-flow-b548436559 branch 2 times, most recently from 7a24d03 to 2139300 Compare September 7, 2026 14:20
@VLukiantsev
VLukiantsev marked this pull request as ready for review September 7, 2026 14:31
@VLukiantsev
VLukiantsev force-pushed the feat/components-health-status-flow-b548436559 branch 3 times, most recently from dfd60d4 to 9111563 Compare September 7, 2026 16:57

@andrzejfudro andrzejfudro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few minor things.

Just a note, I will prepare follow up to this PR that will unify spanner connections (custom components can reuse the one that we initialize for ess db check) and add lazy reinitialization for custom components provider if it fails on the start. I don't want to block this PR on that because it is not that straightforward so I'll take care of that part.

Comment thread cmd/health_service/health_checker.go Resolved
Comment thread cmd/health_service/main.go Resolved
Comment thread cmd/health_service/health_checker.go Resolved
Comment thread cmd/health_service/server.go Resolved
Comment thread cmd/health_service/health_checker.go Resolved
Comment thread cmd/health_service/main.go Resolved
…rting

Introduces pkg/components/summary CustomComponentsInfoProvider library
querying component state directly from the Spanner Store and integrates
operational health evaluation into cmd/health_service GetHealthStatus.

Fixes b/548436559

TAG=agy
CONV=2181912f-1e25-4102-af4e-a8ae3e9b2195
…nents and fail fast on Spanner init

- Update calculateHealthStatus to treat non-healthy core services including UNSPECIFIED as UNHEALTHY, while ignoring UNSPECIFIED only for custom components
- Fail fast with logging.Fatal during health_service startup when EnableComponents is true and Spanner client or StoreComponentsProvider creation fails
- Add unit test verifying UNSPECIFIED health state isolation across core services vs custom components

TAG=agy
CONV=2181912f-1e25-4102-af4e-a8ae3e9b2195
…nd defensive nil checks

- Add 3-second timeout for GetComponentsHealth in HealthChecker.RunChecks
- Add defensive nil check for components in calculateHealthStatus
- Add unit test verifying nil component safety in calculateHealthStatus

TAG=agy
CONV=2181912f-1e25-4102-af4e-a8ae3e9b2195
- Add pkg/components/summary/README.md detailing architecture, SOT invariants, defensive timeout, and operational health evaluation contracts

TAG=agy
CONV=2181912f-1e25-4102-af4e-a8ae3e9b2195
- Add enable_components to health-config ConfigMap mapped from componentsDeployment.enabled
- Bind enable_components env var in health_service config

TAG=agy
CONV=2181912f-1e25-4102-af4e-a8ae3e9b2195
…reporting, and lifecycle

Comment link: #792 (comment)
Comment link: #792 (comment)
Comment link: #792 (comment)
Comment link: #792 (comment)
Comment link: #792 (comment)
Comment link: #792 (comment)

TAG=agy
CONV=2181912f-1e25-4102-af4e-a8ae3e9b2195
@VLukiantsev
VLukiantsev force-pushed the feat/components-health-status-flow-b548436559 branch from 9111563 to 5a24588 Compare September 9, 2026 07:18
@VLukiantsev
VLukiantsev removed this pull request from stack #794 September 9, 2026 07:36
@VLukiantsev
VLukiantsev added this pull request to stack #828 September 9, 2026 07:37
@VLukiantsev
VLukiantsev added this pull request to the merge queue Sep 9, 2026
Merged via the queue into main with commit 3ab6436 Sep 9, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants