Repository navigation
feat(actuator): integrate component snapshot discovery and placement gating - #802
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces runtime component snapshot injection and placement gating into the XMC Actuator daemon. It enables the actuator to parse component snapshots, query component custom resources using a cached controller-runtime client (with dynamic client and cluster-wide fallbacks), evaluate placement readiness (enablement, health, and generation drift), and inject access configurations (environment variables) into JobSet containers. Additionally, error handling in the task handler has been updated to correctly detect wrapped retryable API errors. Feedback on the changes suggests explicitly validating that the status map is present and non-nil before accessing its keys in extractObservedGeneration and similar helper functions to improve robustness and readability.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request implements component snapshot injection and placement gating within the actuator daemon, allowing JobSets to resolve and inject configurations from Component custom resources while enforcing health and enablement constraints. The review feedback highlights several critical robustness improvements for the injection logic in component_injection.go: handling json.Number in extractObservedGeneration to avoid false-positive drift warnings, converting non-string scalar values to strings in extractStringMap to prevent environment variables from being silently dropped, and requiring a Ready condition in evaluateComponentHealth to avoid scheduling workloads on uninitialized components.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request implements component snapshot injection and placement gating in the actuator daemon, allowing JobSets to consume locked component configurations and inject their environment variables into primary and init containers. It also updates the task handler's retry logic to support wrapped API errors. The review feedback highlights two critical issues: first, the controller-runtime client is initialized as a direct client rather than a cached reader, which contradicts the documentation and could overload the API server under high load; second, the isRetryableError helper should explicitly handle context.DeadlineExceeded to ensure context timeouts are correctly retried.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request implements component snapshot injection and placement gating within the actuator daemon. It introduces logic to fetch Component custom resources, validate their health and enablement status, detect generation drift, and inject their access environment variables into synthesized JobSets. Additionally, the task handler is updated to support retries on service-unavailable errors. The review feedback focuses on improving robustness when parsing unstructured Kubernetes objects by replacing manual map type assertions with safer helper functions from the unstructured package, such as NestedBool, NestedString, NestedSlice, NestedFieldNoCopy, and NestedMap.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request implements component snapshot injection and placement gating in the Kubernetes actuator daemon. It enables parsing component snapshots from annotations, retrieving the corresponding Component custom resources, evaluating their health and enablement states, and injecting their access environment variables into the synthesized JobSet containers. Additionally, error handling is updated to correctly identify wrapped retryable errors. The review feedback identifies a potential issue in the health evaluation logic, suggesting that any Ready condition status other than 'True' should be treated as unhealthy to prevent uninitialized states from being incorrectly considered healthy.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request implements runtime component snapshot injection and placement gating in the XMC Actuator Daemon. It introduces the capability to resolve Component custom resources via controller-runtime, validate their health and enablement, detect generation drift, and inject access configurations (environment variables) into JobSet containers. It also updates error handling in the task handler to support wrapped retryable errors. Feedback was provided to improve the detection of wrapped network timeouts and temporary errors in isRetryableError using errors.As with interface targets.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces runtime component snapshot injection and placement gating in the Actuator daemon. It adds the capability to fetch Component custom resources (using controller-runtime with a cluster-wide fallback), validate their health and enablement status, detect generation drift, and inject access configurations (environment variables) into JobSet containers. Additionally, error handling in the task handler was improved to correctly unwrap and identify retryable errors. The review feedback suggests enhancing error handling within the extraction of component access configurations to ensure that malformed statuses fail fast rather than being silently ignored.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces component snapshot injection into the XMC Actuator daemon, allowing it to resolve component configurations from Kubernetes Component custom resources rather than querying Cloud Spanner directly. Key changes include the implementation of component discovery, placement gating, drift detection, and environment variable injection into JobSets. Additionally, the isRetryableError logic in the task handler has been expanded to support more error types, and comprehensive tests have been added to verify these new capabilities. I have no feedback to provide.
AntonTyb
left a comment
There was a problem hiding this comment.
one error message nitpick
619ef5a to
c631228
Compare
c631228 to
54a62cc
Compare
54a62cc to
4c5aa54
Compare
…gating Integrate Component snapshot discovery and placement gating into Actuator JobSet synthesis. Read component snapshot annotations, evaluate placement enablement and health, detect generation drift, and gate placement using retryable errors for hydration lag and health failures. TAG=agy CONV=a22ce4d3-ff57-4515-bf2f-e810b9aec5ce
4c5aa54 to
082705d
Compare
Summary
Integrates component snapshot discovery, feature enablement gating, and conditional placement gating into Actuator JobSet synthesis. Actuator reads required component snapshots from the
xmc.dev/component-snapshotsJobSet annotation, respects cluster feature enablement (enable_components), resolves Component CRs viacontroller-runtimecache with cluster-wide fallback, gates workload placement by failing fast on missing/unhealthy/disabled components with retryable errors, and emits drift warnings on generation mismatches without blocking placement.Motivation & Context
WorkUnits require access to platform-managed components (such as storage drivers, monitoring agents, or credentials). Per the AI Components Architecture, Actuator must discover and validate required component readiness during JobSet synthesis without querying Cloud Spanner directly. In addition, WorkUnits must not be scheduled if required components are degraded, disabled, or undergoing initial CR hydration lag. When components are disabled cluster-wide (
enable_components=false), requests requiring components fail fast immediately rather than looping indefinitely in Redis retries.This is Layer 1 of a 3-PR stack (
gh-stack):status.access.envVarsand secret referencesstatus.access.secretEnvVarsviacorev1.SecretKeyRef).status.access.secretVolumeMounts) with RFC 1123 DNS subdomain sanitization and collision deduplication.Key Changes
enable_components: {{ .Values.componentsDeployment.enabled | quote }}to Actuator ConfigMap (deploy/helm/templates/actuator/configmap.yaml).deploy/helm/templates/actuator/clusterrole.yamlforxmc.dev/components(get,list,watch).--enable_componentsCLI flag andenable_componentsconfig env incmd/actuator/config.goandk8sclient.ActuatorRuntimeConfig.evaluateComponentSnapshots, fails fast with a non-retryable error if component snapshots are requested on a cluster withenable_components=false.ComponentSnapshotsAnnotation(xmc.dev/component-snapshots) andComponentDriftWarningAnnotation(xmc.dev/component-drift-warning) ininternal/kubernetes/common/constants/constants.go.cmd/actuator/k8s_client/component_injection.go, implementedgetComponentusingc.ManagerClient(runtimeclient.Reader) falling back todynamicClientand the cluster-wide"xmc"namespace.apierrors.NewServiceUnavailableto trigger task requeue with backoff.spec.enabled == falseor terminal status): Returnsapierrors.NewServiceUnavailable.Ready=False/Unknown/PendingorDegraded=True): Returnsapierrors.NewServiceUnavailable.observedGeneration, logging a warning and appendingxmc.dev/component-drift-warningwithout blocking placement.isRetryableErrorwitherrors.As) into preparatory PR fix(actuator): unwrap and classify retryable errors in workunit task handler #814 to enforce strict Single Responsibility Principle between worker loop and K8s JobSet construction.cmd/actuator/README.mdand updatedpkg/components/README.mdwith runtime discovery and gating architecture.Assumptions
xmc.dev/component-snapshotsannotation on the JobSet specification as a JSON map of component names to locked generations.controller-runtimecache is initialized viac.ManagerClienton the actuator client; fallback to dynamic client is preserved for unit tests.MaxRestarts: 0,restartPolicy: Never/OnFailure) remains strictly enforced.Verification & Testing
bazel test //cmd/actuator/...executed and passed 100%://cmd/actuator:actuator_test(PASSED)//cmd/actuator/k8s_client:k8s_client_test(PASSED, 10/10 shards)//cmd/actuator/k8s_client/preconditions_validator:preconditions_validator_test(PASSED)//cmd/actuator/workunittaskhandler:workunittaskhandler_test(PASSED)component_injection_test.gocovering healthy admission, feature disabled with/without snapshots, missing CR hydration lag, disabled CR states, unhealthy condition states (Ready=False/Unknown/Pending,Degraded=True, uninitialized/empty conditions), generation drift detection (single and multiple), locked generation <= 0, malformed annotation JSON, canceled context during fetch, and cluster fallback.component_injection_test.goto maintain Single Responsibility Principle../.githooks/pre-commitpassed cleanly (buildifier, gofmt, golangci-lint, determinism, addlicense).TAG=agy
CONV=a22ce4d3-ff57-4515-bf2f-e810b9aec5ce