Repository navigation
Tags
Tags
docs(deploy): document CTK Terraform state storage outside repository… … worktrees (#1686) * docs(deploy): document CTK Terraform state storage outside repository worktrees ## Summary Document Terraform state (`terraform.tfstate`) storage best practices for Cluster Toolkit v2 (`deploy/ctk_v2`) deployments, explicitly warning against storing `terraform.tfstate` files inside any Git repository checkout or linked `git worktree` and listing recommended storage options outside the repository. ## Motivation & Context By default, running `gcluster create ... -o ctk-out` without a configured backend writes `terraform.tfstate` files under `<repo>/ctk-out/<deployment_name>/<group>/terraform.tfstate`. Storing infrastructure state inside a Git checkout or worktree mixes application source code with live infrastructure state—two concerns with independent lifecycles that should be versioned separately. Because `/ctk-out` is gitignored, running `gcluster deploy` from an isolated `git worktree` starts with empty state and fails with GCP `409 Conflict` (`already exists`) errors, while removing a worktree (`git worktree remove`) permanently deletes any state stored inside it. ## Key Changes - `deploy/ctk_v2/README.md`: Add the **Terraform State Storage (`terraform.tfstate`)** section warning against in-repo state files across worktrees and comparing storage options: 1. Google Cloud Storage (`gcs` backend via `terraform_backend_defaults`) with GCS Object Versioning for shared, multi-worktree, and CI/production environments. 2. Dedicated external host directory outside all Git worktrees (e.g. `~/.local/share/xmc/ctk-out`) for single-developer personal environments. 3. Enterprise remote backends (`http`, `pg`, etc.) via `terraform_backend_defaults`. - `deploy/ctk_v2/xmc-deployment-template.yaml`: Add a commented `terraform_backend_defaults` (`type: gcs`) configuration block referencing the state storage guide. - `INSTALL.md` & `tools/dev_env/README.md`: Add cross-reference links to the canonical Terraform state storage section in `deploy/ctk_v2/README.md`. ## Assumptions None — requirements were fully specified. ## Verification & Testing - **Commands Executed**: `./.githooks/pre-commit`, `./tools/analyze_diff_size.py --base-ref origin/main...` - **Coverage & Test Status**: All pre-commit linters (`check_agent_doc_paths.py`, Trivy config scan, legacy import ratchet, and diff sizing check) passed with zero errors. ## Related Issues / Bugs Fixes bug 565291758 TAG=agy CONV=896b370e-05f3-428e-a631-bc24d7b4707b Requested-by: grain Approved-by: grain * docs(deploy): address PR #1686 review comments on CTK state setup - Remove any existing ./ctk-out directory (after migrating prior state if applicable) before creating the external directory symlink so `ln -s` does not create a nested `ctk-out/ctk-out` link. - Add an inline comment noting that `gcloud storage buckets create` does not accept `--versioning` and requires `gcloud storage buckets update --versioning`. Comment link: #1686 (comment) Comment link: #1686 (comment) TAG=agy CONV=896b370e-05f3-428e-a631-bc24d7b4707b Requested-by: grain Approved-by: grain * docs(deploy): address PR #1686 review feedback on state migration and worktrees - Document GCS state prefix (`xmanager-on-cloud-v2/<deployment_name>/<group>`), `deployment_name` uniqueness per bucket, `terraform init -migrate-state -force-copy` migration from local state, and new-worktree setup (copying the gitignored deployment YAML and compiling `ctk-out`). - Guard the local `./ctk-out` directory replacement with `cp -a ctk-out/. "${HOME}/.local/share/xmc/ctk-out/" && rm -rf ctk-out` before `ln -sfn` so existing local `terraform.tfstate` files are preserved. - Document Variant A vs. Variant B path requirements for `tools/dev_env` and Phase 2 commands, along with shared external directory caveats (no `gcluster create` lock across worktrees and single-copy `.backup` files). - Clarify the `xmc-deployment-template.yaml` comment on migrating existing local state first. - Move the `terraform.tfstate` callout in `tools/dev_env/README.md` into a dedicated `> [!IMPORTANT]` block. Comment link: #1686 (comment) Comment link: #1686 (comment) Comment link: #1686 (comment) Comment link: #1686 (comment) Comment link: #1686 (comment) Comment link: #1686 (comment) Comment link: #1686 (comment) TAG=agy CONV=896b370e-05f3-428e-a631-bc24d7b4707b Requested-by: grain Approved-by: grain --------- Co-authored-by: AntonTyb <29805012+AntonTyb@users.noreply.github.com>
docs(deploy): document CTK Terraform state storage outside repository… … worktrees (#1686) * docs(deploy): document CTK Terraform state storage outside repository worktrees ## Summary Document Terraform state (`terraform.tfstate`) storage best practices for Cluster Toolkit v2 (`deploy/ctk_v2`) deployments, explicitly warning against storing `terraform.tfstate` files inside any Git repository checkout or linked `git worktree` and listing recommended storage options outside the repository. ## Motivation & Context By default, running `gcluster create ... -o ctk-out` without a configured backend writes `terraform.tfstate` files under `<repo>/ctk-out/<deployment_name>/<group>/terraform.tfstate`. Storing infrastructure state inside a Git checkout or worktree mixes application source code with live infrastructure state—two concerns with independent lifecycles that should be versioned separately. Because `/ctk-out` is gitignored, running `gcluster deploy` from an isolated `git worktree` starts with empty state and fails with GCP `409 Conflict` (`already exists`) errors, while removing a worktree (`git worktree remove`) permanently deletes any state stored inside it. ## Key Changes - `deploy/ctk_v2/README.md`: Add the **Terraform State Storage (`terraform.tfstate`)** section warning against in-repo state files across worktrees and comparing storage options: 1. Google Cloud Storage (`gcs` backend via `terraform_backend_defaults`) with GCS Object Versioning for shared, multi-worktree, and CI/production environments. 2. Dedicated external host directory outside all Git worktrees (e.g. `~/.local/share/xmc/ctk-out`) for single-developer personal environments. 3. Enterprise remote backends (`http`, `pg`, etc.) via `terraform_backend_defaults`. - `deploy/ctk_v2/xmc-deployment-template.yaml`: Add a commented `terraform_backend_defaults` (`type: gcs`) configuration block referencing the state storage guide. - `INSTALL.md` & `tools/dev_env/README.md`: Add cross-reference links to the canonical Terraform state storage section in `deploy/ctk_v2/README.md`. ## Assumptions None — requirements were fully specified. ## Verification & Testing - **Commands Executed**: `./.githooks/pre-commit`, `./tools/analyze_diff_size.py --base-ref origin/main...` - **Coverage & Test Status**: All pre-commit linters (`check_agent_doc_paths.py`, Trivy config scan, legacy import ratchet, and diff sizing check) passed with zero errors. ## Related Issues / Bugs Fixes bug 565291758 TAG=agy CONV=896b370e-05f3-428e-a631-bc24d7b4707b Requested-by: grain Approved-by: grain * docs(deploy): address PR #1686 review comments on CTK state setup - Remove any existing ./ctk-out directory (after migrating prior state if applicable) before creating the external directory symlink so `ln -s` does not create a nested `ctk-out/ctk-out` link. - Add an inline comment noting that `gcloud storage buckets create` does not accept `--versioning` and requires `gcloud storage buckets update --versioning`. Comment link: #1686 (comment) Comment link: #1686 (comment) TAG=agy CONV=896b370e-05f3-428e-a631-bc24d7b4707b Requested-by: grain Approved-by: grain * docs(deploy): address PR #1686 review feedback on state migration and worktrees - Document GCS state prefix (`xmanager-on-cloud-v2/<deployment_name>/<group>`), `deployment_name` uniqueness per bucket, `terraform init -migrate-state -force-copy` migration from local state, and new-worktree setup (copying the gitignored deployment YAML and compiling `ctk-out`). - Guard the local `./ctk-out` directory replacement with `cp -a ctk-out/. "${HOME}/.local/share/xmc/ctk-out/" && rm -rf ctk-out` before `ln -sfn` so existing local `terraform.tfstate` files are preserved. - Document Variant A vs. Variant B path requirements for `tools/dev_env` and Phase 2 commands, along with shared external directory caveats (no `gcluster create` lock across worktrees and single-copy `.backup` files). - Clarify the `xmc-deployment-template.yaml` comment on migrating existing local state first. - Move the `terraform.tfstate` callout in `tools/dev_env/README.md` into a dedicated `> [!IMPORTANT]` block. Comment link: #1686 (comment) Comment link: #1686 (comment) Comment link: #1686 (comment) Comment link: #1686 (comment) Comment link: #1686 (comment) Comment link: #1686 (comment) Comment link: #1686 (comment) TAG=agy CONV=896b370e-05f3-428e-a631-bc24d7b4707b Requested-by: grain Approved-by: grain --------- Co-authored-by: AntonTyb <29805012+AntonTyb@users.noreply.github.com>
docs(deploy): document CTK Terraform state storage outside repository… … worktrees (#1686) * docs(deploy): document CTK Terraform state storage outside repository worktrees ## Summary Document Terraform state (`terraform.tfstate`) storage best practices for Cluster Toolkit v2 (`deploy/ctk_v2`) deployments, explicitly warning against storing `terraform.tfstate` files inside any Git repository checkout or linked `git worktree` and listing recommended storage options outside the repository. ## Motivation & Context By default, running `gcluster create ... -o ctk-out` without a configured backend writes `terraform.tfstate` files under `<repo>/ctk-out/<deployment_name>/<group>/terraform.tfstate`. Storing infrastructure state inside a Git checkout or worktree mixes application source code with live infrastructure state—two concerns with independent lifecycles that should be versioned separately. Because `/ctk-out` is gitignored, running `gcluster deploy` from an isolated `git worktree` starts with empty state and fails with GCP `409 Conflict` (`already exists`) errors, while removing a worktree (`git worktree remove`) permanently deletes any state stored inside it. ## Key Changes - `deploy/ctk_v2/README.md`: Add the **Terraform State Storage (`terraform.tfstate`)** section warning against in-repo state files across worktrees and comparing storage options: 1. Google Cloud Storage (`gcs` backend via `terraform_backend_defaults`) with GCS Object Versioning for shared, multi-worktree, and CI/production environments. 2. Dedicated external host directory outside all Git worktrees (e.g. `~/.local/share/xmc/ctk-out`) for single-developer personal environments. 3. Enterprise remote backends (`http`, `pg`, etc.) via `terraform_backend_defaults`. - `deploy/ctk_v2/xmc-deployment-template.yaml`: Add a commented `terraform_backend_defaults` (`type: gcs`) configuration block referencing the state storage guide. - `INSTALL.md` & `tools/dev_env/README.md`: Add cross-reference links to the canonical Terraform state storage section in `deploy/ctk_v2/README.md`. ## Assumptions None — requirements were fully specified. ## Verification & Testing - **Commands Executed**: `./.githooks/pre-commit`, `./tools/analyze_diff_size.py --base-ref origin/main...` - **Coverage & Test Status**: All pre-commit linters (`check_agent_doc_paths.py`, Trivy config scan, legacy import ratchet, and diff sizing check) passed with zero errors. ## Related Issues / Bugs Fixes bug 565291758 TAG=agy CONV=896b370e-05f3-428e-a631-bc24d7b4707b Requested-by: grain Approved-by: grain * docs(deploy): address PR #1686 review comments on CTK state setup - Remove any existing ./ctk-out directory (after migrating prior state if applicable) before creating the external directory symlink so `ln -s` does not create a nested `ctk-out/ctk-out` link. - Add an inline comment noting that `gcloud storage buckets create` does not accept `--versioning` and requires `gcloud storage buckets update --versioning`. Comment link: #1686 (comment) Comment link: #1686 (comment) TAG=agy CONV=896b370e-05f3-428e-a631-bc24d7b4707b Requested-by: grain Approved-by: grain * docs(deploy): address PR #1686 review feedback on state migration and worktrees - Document GCS state prefix (`xmanager-on-cloud-v2/<deployment_name>/<group>`), `deployment_name` uniqueness per bucket, `terraform init -migrate-state -force-copy` migration from local state, and new-worktree setup (copying the gitignored deployment YAML and compiling `ctk-out`). - Guard the local `./ctk-out` directory replacement with `cp -a ctk-out/. "${HOME}/.local/share/xmc/ctk-out/" && rm -rf ctk-out` before `ln -sfn` so existing local `terraform.tfstate` files are preserved. - Document Variant A vs. Variant B path requirements for `tools/dev_env` and Phase 2 commands, along with shared external directory caveats (no `gcluster create` lock across worktrees and single-copy `.backup` files). - Clarify the `xmc-deployment-template.yaml` comment on migrating existing local state first. - Move the `terraform.tfstate` callout in `tools/dev_env/README.md` into a dedicated `> [!IMPORTANT]` block. Comment link: #1686 (comment) Comment link: #1686 (comment) Comment link: #1686 (comment) Comment link: #1686 (comment) Comment link: #1686 (comment) Comment link: #1686 (comment) Comment link: #1686 (comment) TAG=agy CONV=896b370e-05f3-428e-a631-bc24d7b4707b Requested-by: grain Approved-by: grain --------- Co-authored-by: AntonTyb <29805012+AntonTyb@users.noreply.github.com>
ci(workflows): migrate GitHub Actions workflows from arc-runner-set t… …o ubuntu-latest (#1680) * ci(workflows): migrate GitHub Actions workflows from arc-runner-set to ubuntu-latest Migrate all remaining GitHub Actions workflows (16 jobs across 11 workflow files) from self-hosted GKE Actions Runner Controller (arc-runner-set) to standard GitHub-hosted runners (ubuntu-latest) with Bazel disk caching via actions/cache and Playwright compatibility on Ubuntu 26.04. - Runner Migration: Update runs-on to 'ubuntu-latest' across all workflows. - Bazel Caching: Configure actions/cache for ~/.cache/bazel-repo and ~/.cache/bazel-disk with --disk_cache, --remote_cache_compression, and GoLink no-remote-cache. - Playwright Ubuntu 26.04 Compatibility: Update Playwright to 1.61.1 and provide PLAYWRIGHT_HOST_PLATFORM_OVERRIDE=ubuntu24.04-x64 across nightly E2E, visual snapshots, and UI unit test suites, resolving the failing setup in run 43223267217. - E2E Harness Synchronization: Align timeouts to 45 minutes and retry/deadline budgets. - Test Fixes: Set SPANNER_EMULATOR_HOST in actuator prod_test for non-GCP runners. - Documentation: Update docs/ci_runner_infrastructure.md and align cache save conditions. TAG=agy CONV=bec5aef6-400b-476c-8045-4923a3f8ceaf * fix(ci): pass expanded HOME disk_cache path and fix cache-matched-key output Address review feedback on PR #1680: - Remove unexpanded '~' --disk_cache from repo .bazelrc and pass --disk_cache=${HOME}/.cache/bazel-disk from CI workflows (run-backend-tests, e2e-tests, deploy-gke), matching --repository_cache=${HOME}/.cache/bazel-repo and keeping ~/.cache/bazel-disk in actions/cache without workspace root pollution. - Fix actions/cache/restore output reference from matched-key to cache-matched-key in run-backend-tests/action.yml, e2e-tests.yml, and docs/ci_runner_infrastructure.md so PR runs do not unconditionally save caches. - Synchronize docs/ci_runner_infrastructure.md with the updated cache configuration. TAG=agy CONV=bec5aef6-400b-476c-8045-4923a3f8ceaf * fix(ci): pin Python playwright==1.61.0 in nightly workflows Python playwright PyPI releases jumped from 1.61.0 to 1.62.0 (1.61.1 only exists for npm @playwright/test), causing pip install to fail in nightly runs 43406478492 and 43406484201. TAG=agy CONV=bec5aef6-400b-476c-8045-4923a3f8ceaf * fix(ci): install gke-gcloud-auth-plugin via setup-gcloud in deploy-gke.yml ubuntu-latest (Ubuntu 26.04) has gcloud component manager disabled and lacks google-cloud-cli-gke-gcloud-auth-plugin in apt, causing kubectl calls in deploy-gke.yml (run 43410623242) to fail. Use google-github-actions/setup-gcloud@v3.0.1 with install_components: gke-gcloud-auth-plugin and fail fast if the plugin binary is missing. TAG=agy CONV=bec5aef6-400b-476c-8045-4923a3f8ceaf * fix(ci): ensure helm v3.20.1 is installed on ubuntu-latest in deploy-gke.yml ubuntu-latest pre-installs Helm v4.3.1, so 'command -v helm' skipped installing helm-v3.20.1. Helm v4 --wait uses kstatus wait by default and times out on disabled Component CRs (Component/xmc/vizier, Component/xmc/hpo) during xmc-profile deployment (run 43411519735). Check 'helm version --short | grep -q ^v3\.' so v3.20.1 is installed into ~/.local/bin. TAG=agy CONV=bec5aef6-400b-476c-8045-4923a3f8ceaf
fix(ci): ensure helm v3.20.1 is installed on ubuntu-latest in deploy-… …gke.yml ubuntu-latest pre-installs Helm v4.3.1, so 'command -v helm' skipped installing helm-v3.20.1. Helm v4 --wait uses kstatus wait by default and times out on disabled Component CRs (Component/xmc/vizier, Component/xmc/hpo) during xmc-profile deployment (run 43411519735). Check 'helm version --short | grep -q ^v3\.' so v3.20.1 is installed into ~/.local/bin. TAG=agy CONV=bec5aef6-400b-476c-8045-4923a3f8ceaf
fix(ci): install gke-gcloud-auth-plugin via setup-gcloud in deploy-gk… …e.yml ubuntu-latest (Ubuntu 26.04) has gcloud component manager disabled and lacks google-cloud-cli-gke-gcloud-auth-plugin in apt, causing kubectl calls in deploy-gke.yml (run 43410623242) to fail. Use google-github-actions/setup-gcloud@v3.0.1 with install_components: gke-gcloud-auth-plugin and fail fast if the plugin binary is missing. TAG=agy CONV=bec5aef6-400b-476c-8045-4923a3f8ceaf
fix(ci): pin Python playwright==1.61.0 in nightly workflows Python playwright PyPI releases jumped from 1.61.0 to 1.62.0 (1.61.1 only exists for npm @playwright/test), causing pip install to fail in nightly runs 43406478492 and 43406484201. TAG=agy CONV=bec5aef6-400b-476c-8045-4923a3f8ceaf
fix(nightly): resolve WorkUnitState names from status_pb2 with synced… … fallback (#1652) * fix(nightly): resolve WorkUnitState names from status_pb2 with synced fallback ## Summary Resolve `WorkUnitStatus.WorkUnitState` enum names directly from `status_pb2` in `tools/nightly_tests/verification/xmc_entities_validator.py` when the external `xmanager_cloud` package is installed, and update the static `STATE_NAMES` fallback list to include `SCHEDULING_PAUSED` (`12`) alongside a unit test that verifies `STATE_NAMES` against `api/protos/experiment/status.proto`. ## Motivation & Context `tools/nightly_tests/verification/xmc_entities_validator.py` maintained a hand-copied list of `WorkUnitStatus.WorkUnitState` names (`0..11`) so Bazel unit tests can run hermetically without the external `xmanager_cloud` wheel. When `SCHEDULING_PAUSED = 12` was added to `api/protos/experiment/status.proto`, `state_enum_to_name(12)` returned `"UNKNOWN(12)"` even when `xmanager_cloud` was installed, and no test guarded the fallback list against proto additions. ## Key Changes - Import `status_pb2` alongside `api_pb2` and `priority_pb2` in the optional `xmanager_cloud.experiment_state_server.proto` import block in `tools/nightly_tests/verification/xmc_entities_validator.py`. - Update `state_enum_to_name` to query `status_pb2.WorkUnitStatus.WorkUnitState.Name(state_enum)` (stripping the `WORK_UNIT_STATE_` prefix on `WORK_UNIT_STATE_UNSPECIFIED`) when `status_pb2` is available, falling back to `STATE_NAMES` only when `status_pb2` is `None`. - Add `"SCHEDULING_PAUSED"` (`12`) to `STATE_NAMES` and export `STATE_SCHEDULING_PAUSED` in `tools/nightly_tests/verification/xmc_entities_validator.py`. - Export `status.proto` in `api/protos/experiment/BUILD.bazel`, attach `data = ["//api/protos/experiment:status.proto"]` to `xmc_entities_validator_test` in `tools/nightly_tests/verification/BUILD.bazel`, and add unit tests in `tools/nightly_tests/verification/xmc_entities_validator_test.py` covering `status_pb2` resolution, fallback lookup, and exact parity with `status.proto`. ## Assumptions None — requirements were fully specified. ## Verification & Testing - **Commands Executed**: `bazel test //tools/nightly_tests/verification:xmc_entities_validator_test` - **Coverage & Test Status**: All 69 unit tests in `XmcEntitiesValidatorTest` (including `test_state_enum_to_name`, `test_state_enum_to_name_with_proto_enum`, and `test_state_names_fallback_matches_status_proto`) pass. ## Related Issues / Bugs Fixes bug b/569895806 Bug: b/569895806 TAG=agy CONV=64ce2f30-ad8b-49fd-b7bc-e1a0c3311376 Requested-by: vluk@google.com Approved-by: vluk@google.com * fix(nightly): fall back to STATE_NAMES on status_pb2 AttributeError or ValueError Catch `(AttributeError, ValueError)` in `state_enum_to_name` and fall through to `STATE_NAMES` so older `xmanager_cloud` wheels lacking newer enum values still resolve from the synced fallback list, and update `test_state_enum_to_name_with_proto_enum` accordingly. Comment link: #1652 (comment) Comment link: #1652 (comment) Bug: b/569895806 TAG=agy CONV=64ce2f30-ad8b-49fd-b7bc-e1a0c3311376 Requested-by: vluk@google.com Approved-by: vluk@google.com --------- Co-authored-by: VLukiantsev <23403984+VLukiantsev@users.noreply.github.com>
fix(actuator): use the gcsfuse.csi.storage.gke.io CSI driver for GCS … …volumes (#1586) * fix(actuator): use the gcsfuse.csi.storage.gke.io CSI driver for GCS volumes GcsVolumeSource was rendered as a CSI volume with driver gcs.csi.storage.gke.io, which GKE does not register. Pods using a GCS volume stayed in ContainerCreating. The GKE GCS FUSE CSI driver is gcsfuse.csi.storage.gke.io; use it in the actuator's volume builder and in the GCS sidecar annotation check, and update the proto comment and tests. * fix(actuator): annotate GCS FUSE pods only from job GCS volumes The pod-spec fallback in annotateGCSVolumes could only match a CSI volume that did not come from job.Volumes[].gcs, and the actuator never builds one: generateVolumesForJob creates CSI volumes only from Gcs sources, and component injection adds only Secret volumes after InjectOnJobSetCreation. Removing it also removes the second copy of the driver name. * fix(deploy): enable the GCS FUSE CSI driver in the CTK v2 blueprint GcsVolumeSource volumes use the gcsfuse.csi.storage.gke.io driver. The Cluster Toolkit gke-cluster module defaults enable_gcsfuse_csi to false, so on a cluster built from this blueprint the driver and its sidecar injection webhook are missing, and pods with a GCS volume stay in ContainerCreating. * docs: document GCS FUSE CSI driver prerequisites for GCS volumes GCS volumes need the Cloud Storage FUSE CSI driver add-on on the cluster and a Workload Identity binding for the job's Kubernetes service account. The CTK v2 blueprint now enables the add-on, but clusters provisioned directly with Terraform (Approach 3) do not get it. SDK users read the GcsVolumeSource comment when they set the field, so document both prerequisites there.
PreviousNext