Repository navigation
feat(components): implement custom components operational health reporting - #792
Conversation
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request integrates custom components discovery and health checking into the health service, introducing a new summary package to retrieve and evaluate custom component health states. The health checker is updated to run these checks in parallel with existing Spanner checks. The review feedback highlights two important issues: first, changing the overall health check logic to only trigger on HEALTH_STATE_UNHEALTHY risks masking core service failures that report as UNSPECIFIED, so UNSPECIFIED states should only be ignored for custom components. Second, if custom components are enabled, initialization failures for the Spanner client or store provider should result in a fatal error rather than silently disabling the provider.
There was a problem hiding this comment.
Code Review
This pull request introduces custom components discovery and health checking to the health service, allowing it to query and report the health of custom 1P and 3P components alongside core services. The feedback focuses on improving robustness: first, by adding a dedicated 3-second timeout to the custom components health check to prevent slow providers from delaying the entire health check response; and second, by adding defensive nil checks in the health status calculation loop to avoid potential nil-pointer dereference panics.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request integrates custom components discovery and health checking into the health service. It introduces a new summary package containing StoreComponentsProvider to retrieve and evaluate custom component health statuses from a backing store. The health checker is updated to query these custom components in parallel with core Spanner checks, and the server is adjusted to prevent disabled custom components from degrading the overall health status. Feedback suggests adding a nil check for components in calculateHealthStatus to avoid potential nil pointer dereferences.
7a24d03 to
2139300
Compare
dfd60d4 to
9111563
Compare
andrzejfudro
left a comment
There was a problem hiding this comment.
Few minor things.
Just a note, I will prepare follow up to this PR that will unify spanner connections (custom components can reuse the one that we initialize for ess db check) and add lazy reinitialization for custom components provider if it fails on the start. I don't want to block this PR on that because it is not that straightforward so I'll take care of that part.
…rting Introduces pkg/components/summary CustomComponentsInfoProvider library querying component state directly from the Spanner Store and integrates operational health evaluation into cmd/health_service GetHealthStatus. Fixes b/548436559 TAG=agy CONV=2181912f-1e25-4102-af4e-a8ae3e9b2195
…nents and fail fast on Spanner init - Update calculateHealthStatus to treat non-healthy core services including UNSPECIFIED as UNHEALTHY, while ignoring UNSPECIFIED only for custom components - Fail fast with logging.Fatal during health_service startup when EnableComponents is true and Spanner client or StoreComponentsProvider creation fails - Add unit test verifying UNSPECIFIED health state isolation across core services vs custom components TAG=agy CONV=2181912f-1e25-4102-af4e-a8ae3e9b2195
…nd defensive nil checks - Add 3-second timeout for GetComponentsHealth in HealthChecker.RunChecks - Add defensive nil check for components in calculateHealthStatus - Add unit test verifying nil component safety in calculateHealthStatus TAG=agy CONV=2181912f-1e25-4102-af4e-a8ae3e9b2195
- Add pkg/components/summary/README.md detailing architecture, SOT invariants, defensive timeout, and operational health evaluation contracts TAG=agy CONV=2181912f-1e25-4102-af4e-a8ae3e9b2195
- Add enable_components to health-config ConfigMap mapped from componentsDeployment.enabled - Bind enable_components env var in health_service config TAG=agy CONV=2181912f-1e25-4102-af4e-a8ae3e9b2195
…reporting, and lifecycle Comment link: #792 (comment) Comment link: #792 (comment) Comment link: #792 (comment) Comment link: #792 (comment) Comment link: #792 (comment) Comment link: #792 (comment) TAG=agy CONV=2181912f-1e25-4102-af4e-a8ae3e9b2195
9111563 to
5a24588
Compare
Summary
Layer 1 of the Custom Components Admin Visibility stack (
b/548436559). Introducespkg/components/summaryCustomComponentsInfoProviderlibrary querying component state directly and exclusively from the Spanner Store in-memory, and integrates operational health evaluation intocmd/health_serviceGetHealthStatus.Motivation & Context
The XMC Admin Panel requires visibility into installed Custom Components (1P native services and 3P Kubernetes CRD wrappers). To eliminate external RPC latency bottlenecks and single points of failure on the health check path, this change reads component state exclusively from the Spanner Store (
store.Store.ListComponents).It integrates this provider into
cmd/health_servicebehind the--enable_componentsfeature flag (health_enable_componentsenv var), exposing custom component operational status inGetHealthStatus(ComponentType_COMPONENT_TYPE_CUSTOM_COMPONENT).Key Changes
pkg/components/summary:CustomComponentsInfoProviderinterface andStoreComponentsProviderbacked bystore.Store.GetComponentsHealth), mapping component states (ENABLED,PENDING_RECONCILIATION,DISABLED,DISABLING) and reconciler readiness conditions toHealthStateHealthy,HealthStateUnhealthy, orHealthStateDisabled.String()methods forComponentTypeandHealthStateenums and compile-time interface assertions.cmd/health_service:config.go: AddedEnableComponents boolloaded viaconfigutil.BoolEnvand-enable_componentsflag.health_checker.go: AddedComponentsProvidertoHealthCheckerParams; parallelized custom component health checks insideRunChecksalongside Spanner checks, converting results intoComponentType_COMPONENT_TYPE_CUSTOM_COMPONENTentries. Mapped disabled custom components toHEALTH_STATE_UNSPECIFIED.server.go: Added dependency validation innewServer; updatedcalculateHealthStatusto only flag overall platform health as unhealthy on explicitHEALTH_STATE_UNHEALTHYcomponent states.main.go: Added linear helperinitCustomComponentsProviderwith graceful error recovery; ensured fresh timeout context for deferreddeps.Close.pkg/components/summary/info_provider_test.goandcmd/health_service/health_checker_test.gocovering all positive, error, nil-status, disabled component, and concurrency race detection scenarios.Assumptions
b/548436559) and Prototype PR feat(components): prototype XMC components system with GitOps, CMS, and Spanner SOT #456.Verification & Testing
bazel test //pkg/components/summary:all //cmd/health_service:allpassed cleanly.bazel test --@io_bazel_rules_go//go/config:race --nocache_test_results //cmd/health_service:all //pkg/components/summary:allpassed with 0 data races../.githooks/pre-commit).Related Issues / Bugs
Part of b/548436559
TAG=agy
CONV=2181912f-1e25-4102-af4e-a8ae3e9b2195