From 337da31b400ca921c841e2d0dc2d293af0285574 Mon Sep 17 00:00:00 2001 From: hmmhmmhm Date: Wed, 15 Jul 2026 14:21:10 +0900 Subject: [PATCH] docs: complete r2 region data delivery plan --- docs/operations/region-data-delivery.md | 82 ++++++- .../2026-07-13-r2-region-data-delivery.md | 204 +++++++++--------- 2 files changed, 181 insertions(+), 105 deletions(-) diff --git a/docs/operations/region-data-delivery.md b/docs/operations/region-data-delivery.md index 064dc15d..043c8644 100644 --- a/docs/operations/region-data-delivery.md +++ b/docs/operations/region-data-delivery.md @@ -283,6 +283,12 @@ generation produced the same version, 2,031 entries, and 1,044 unique objects. The tracked pointer remained byte-identical, so the publisher and pull-request paths were not invoked. +The first GitHub-hosted `none` dispatch then completed in [publication run +29389992473][phase-b-publish-none]. It regenerated the same version from all +2,031 entries and 1,044 unique objects, reported `changed=false`, and skipped +the publisher, public re-materialization, documentation, branch, and pull +request steps. GitHub had no `data-release/*` ref or pull request afterward. + For changed content, the workflow publishes immutable objects and the manifest, deletes its local managed directories, reconstructs both groups through the public endpoint, and performs exact manifest verification. It @@ -297,13 +303,65 @@ unrelated source path—makes commit validation fail. A successful run pushes `data-release/` and opens a normal pull request against `main`; the release does not become active until that pull request passes CI and merges. -Before the first content-changing production dispatch, repeat the publisher +Before the first content-changing production dispatch, run the publisher fixture test against a separate non-production bucket and credentials. Require one missing object upload, an immutable manifest written last, a second idempotent publication with zero uploads, and an exact public materialization. Record the bucket, run, and cleanup evidence here without -recording credential values. A no-content `none` dispatch on `main` must also -complete without an R2 publication or pull request before production use. +recording credential values. No content-changing production dispatch occurred +during this cutover; this remains a hard prerequisite for the first one. + +## Phase B stage-gate evidence + +[Pull request 81][phase-b-pr] removed the managed data from Git, proved a +fresh R2-only checkout, and added the protected publisher. Its first browser +smoke attempt exposed Playwright's automatic CI diff capture trying to buffer +the multi-gigabyte deletion. Commit `9c7b601a52a7e95bae997eaa064e4cb66d985b22` +kept commit metadata but disabled diff capture, after which the authoritative +[PR CI][phase-b-pr-ci] passed public materialization, security, formatting, +source-size, runtime-pin, operational, geoint, data, language, lint, type, +package, coverage, build, and browser gates. + +The PR merged as `105334d9b5cec32abc4d68290d6eac46b42d1f6e`. +Its [main CI][phase-b-main-ci] and applicable [Web deployment][phase-b-web-deploy] +passed. The first API deployment stopped before any production mutation: +after syncing only `region-dist`, the tests correctly failed because the +Git-removed embedded `region-db` was absent. [Pull request +82][phase-b-api-hotfix-pr] changed the deploy verifier to materialize both +groups; its [PR CI][phase-b-api-hotfix-ci] passed and it merged as +`c232408950f494bd12bf425f5a3f8c13aa764c7e`. The final [main +CI][phase-b-final-main-ci] and [API deployment][phase-b-api-deploy] both +passed. The API run materialized both groups, passed the previously failing +tests, applied the PostGIS schema, detected no changed dataset, skipped import, +deployed the Worker, and passed commit-pinned production smoke. Grok Spiral +did not run because neither merge changed its path-filtered app, UI, workflow, +or root dependency files. + +An independent [full production smoke][phase-b-production-smoke] checked out +the final main commit and passed readiness, Web metadata, every supported +language profile, API documentation, region search, and error-route checks. +A direct `/metrics` validation reported service `api-ground-codes`, scope +`worker-isolate`, runtime commit +`c232408950f494bd12bf425f5a3f8c13aa764c7e`, and an uptime consistent with the +isolate's `startedAt` timestamp. The API deployment's own smoke also required +that exact runtime commit before succeeding. + +GitHub API re-reads found only +`packages/geoint/region-data-release.json` among the pointer and two managed +tree prefixes. The active `main-protection` ruleset still required strict +`verify` and prevented deletion and non-fast-forward updates. Dependabot +alerts and security updates, secret scanning, and push protection remained +enabled. All check runs on the final merge were green. The publisher +environment still allowed protected branches only, contained exactly the two +R2 write-secret names, and repository workflow permissions remained read-only +by default. + +The final local stage gate began with both managed directories absent. Public +sync downloaded all 2,031 entries and 8,971,117,492 bytes. Independent exact +manifest verification reproduced version +`sha256-4b6d31b92ce300ca4ca6d98fb24d966fad61b098639e51f33134b5922ccd3190`, +901 `region-dist` entries, 1,130 `region-db` entries, and zero missing, extra, +changed, or unreadable paths. The materialization was removed afterward. ### Publication history @@ -336,6 +394,14 @@ A release rollback changes only the committed release pointer in a normal, reviewed pull request. Old immutable manifests and objects remain readable, so restoring the previous pointer does not require an R2 mutation. +At this stage gate, Git history contains only the initial R2 release pointer, +so there is no distinct predecessor to restore. The final clean sync and exact +verification above prove that this retained immutable release remains +readable. After the first content-changing release, rollback means copying the +canonical pointer from the last known-good commit into a new branch, opening a +normal pull request, and requiring CI to materialize and verify that older +release before merge. Never overwrite an object or manifest to roll back. + For an infrastructure rollback, first stop publisher workflows, then revoke the publisher token and remove the two R2 credential secrets. Disable or remove the custom domain only after confirming that no active deployment references @@ -355,3 +421,13 @@ routine rollback step. [phase-a-pr-ci]: https://github.com/hmmhmmhm/ground.codes/actions/runs/29383546050 [phase-a-production-smoke]: https://github.com/hmmhmmhm/ground.codes/actions/runs/29384519200 [phase-a-web-deploy]: https://github.com/hmmhmmhm/ground.codes/actions/runs/29384373723 +[phase-b-api-deploy]: https://github.com/hmmhmmhm/ground.codes/actions/runs/29389806946 +[phase-b-api-hotfix-ci]: https://github.com/hmmhmmhm/ground.codes/actions/runs/29389464495 +[phase-b-api-hotfix-pr]: https://github.com/hmmhmmhm/ground.codes/pull/82 +[phase-b-final-main-ci]: https://github.com/hmmhmmhm/ground.codes/actions/runs/29389806941 +[phase-b-main-ci]: https://github.com/hmmhmmhm/ground.codes/actions/runs/29389253996 +[phase-b-pr]: https://github.com/hmmhmmhm/ground.codes/pull/81 +[phase-b-pr-ci]: https://github.com/hmmhmmhm/ground.codes/actions/runs/29388901071 +[phase-b-production-smoke]: https://github.com/hmmhmmhm/ground.codes/actions/runs/29390670878 +[phase-b-publish-none]: https://github.com/hmmhmmhm/ground.codes/actions/runs/29389992473 +[phase-b-web-deploy]: https://github.com/hmmhmmhm/ground.codes/actions/runs/29389254015 diff --git a/docs/superpowers/plans/2026-07-13-r2-region-data-delivery.md b/docs/superpowers/plans/2026-07-13-r2-region-data-delivery.md index 33f7cf7b..fd25c7b3 100644 --- a/docs/superpowers/plans/2026-07-13-r2-region-data-delivery.md +++ b/docs/superpowers/plans/2026-07-13-r2-region-data-delivery.md @@ -24,8 +24,8 @@ - Modify: `scripts/code-size-policy.test.mjs` - Modify: `package.json` -- [ ] Write tests for stable logical-path ordering, binary-safe SHA-256, uncompressed/compressed sizes, group assignment, duplicate-content object-key reuse, content-derived release version, canonical JSON serialization, and rejection of paths outside `region-dist`/`region-db`. -- [ ] Define this versioned document shape in the tests: +- [x] Write tests for stable logical-path ordering, binary-safe SHA-256, uncompressed/compressed sizes, group assignment, duplicate-content object-key reuse, content-derived release version, canonical JSON serialization, and rejection of paths outside `region-dist`/`region-db`. +- [x] Define this versioned document shape in the tests: ```js { @@ -44,13 +44,13 @@ } ``` -- [ ] Run `node --test scripts/region-data/manifest.test.mjs`; expect RED because the module is absent. -- [ ] Implement deterministic recursive enumeration of regular files only. Reject symlinks, sockets, traversal, absolute paths, duplicate logical paths, unsupported schema versions, and malformed hashes. -- [ ] Compute `version` from canonical entry metadata before release-prefixing object keys so the algorithm has no circular dependency. Do not include timestamps, host paths, usernames, or compression timestamps in the version input. -- [ ] Keep every new maintained module below 450 lines and add the new directory to the source-size boundary test. -- [ ] Expand root `scripts:test` to `node --test scripts/*.test.mjs scripts/region-data/*.test.mjs` so nested region-data contracts are mandatory in CI. -- [ ] Run manifest and source-size tests; expect PASS. -- [ ] Commit as `test(region-data): define immutable manifest contract`. +- [x] Run `node --test scripts/region-data/manifest.test.mjs`; expect RED because the module is absent. +- [x] Implement deterministic recursive enumeration of regular files only. Reject symlinks, sockets, traversal, absolute paths, duplicate logical paths, unsupported schema versions, and malformed hashes. +- [x] Compute `version` from canonical entry metadata before release-prefixing object keys so the algorithm has no circular dependency. Do not include timestamps, host paths, usernames, or compression timestamps in the version input. +- [x] Keep every new maintained module below 450 lines and add the new directory to the source-size boundary test. +- [x] Expand root `scripts:test` to `node --test scripts/*.test.mjs scripts/region-data/*.test.mjs` so nested region-data contracts are mandatory in CI. +- [x] Run manifest and source-size tests; expect PASS. +- [x] Commit as `test(region-data): define immutable manifest contract`. ### Task 2: Generate deterministic compressed release artifacts @@ -62,11 +62,11 @@ - Modify: `package.json` - Modify: `.gitignore` -- [ ] Write tests that generate the same release twice and compare manifest bytes, object bytes, version, and deduplication. Change one source byte and assert only its hash object and release version change. -- [ ] Run the generator tests; expect RED. -- [ ] Gzip every regular file with deterministic metadata and write each unique object once under `.region-data-staging/releases//objects/.json.gz`. The `.json.gz` suffix is retained for the approved object contract even when a `region-db` logical file contains binary LevelDB data. -- [ ] Write `.region-data-staging/releases//manifest.json` only after all objects exist, and calculate a separate SHA-256 over the final manifest bytes. -- [ ] Make the CLI accept only explicit arguments: +- [x] Write tests that generate the same release twice and compare manifest bytes, object bytes, version, and deduplication. Change one source byte and assert only its hash object and release version change. +- [x] Run the generator tests; expect RED. +- [x] Gzip every regular file with deterministic metadata and write each unique object once under `.region-data-staging/releases//objects/.json.gz`. The `.json.gz` suffix is retained for the approved object contract even when a `region-db` logical file contains binary LevelDB data. +- [x] Write `.region-data-staging/releases//manifest.json` only after all objects exist, and calculate a separate SHA-256 over the final manifest bytes. +- [x] Make the CLI accept only explicit arguments: ```sh node scripts/generate-region-data-release.mjs \ @@ -75,7 +75,7 @@ --pointer-out packages/geoint/region-data-release.json ``` -- [ ] The pointer output must be small and canonical: +- [x] The pointer output must be small and canonical: ```js { @@ -85,9 +85,9 @@ } ``` -- [ ] Add `.region-data-staging/` to `.gitignore` and root script `region-data:generate` for the exact CLI above. -- [ ] Run tests; expect PASS. -- [ ] Commit as `feat(region-data): generate content-addressed releases`. +- [x] Add `.region-data-staging/` to `.gitignore` and root script `region-data:generate` for the exact CLI above. +- [x] Run tests; expect PASS. +- [x] Commit as `feat(region-data): generate content-addressed releases`. ### Task 3: Build a fail-closed synchronizer @@ -98,22 +98,22 @@ - Create: `scripts/sync-region-data.mjs` - Modify: `package.json` -- [ ] Test against a local HTTP server for: valid full download, already-valid skip, corrupt-local replacement, manifest hash mismatch, object hash mismatch, truncated gzip, 404, retryable 429/5xx, bounded concurrency, group scope, explicit-path scope, atomic temp rename, default no-prune, and explicit prune. -- [ ] Add traversal and malicious-manifest tests proving no write can escape the requested root. -- [ ] Run `node --test scripts/region-data/sync.test.mjs`; expect RED. -- [ ] Implement pointer validation first, then download `releases//manifest.json` from `REGION_DATA_BASE_URL`, verify `manifestSha256`, validate every entry, and select entries by `--groups` or repeated `--path` flags. -- [ ] For each selected entry, skip an existing matching file; otherwise stream the gzip object to a sibling temporary file, decompress while hashing/counting uncompressed bytes, verify size/SHA-256, fsync/close, then atomically rename. Remove temporary files on all failures. -- [ ] Use default concurrency 4, three attempts, and bounded exponential backoff. Retry network errors, 429, and 5xx only. Fail immediately for other 4xx, schema, hash, size, path, or decompression errors. -- [ ] Default behavior must never delete files. `--prune` removes only files under the selected managed group roots that are absent from the verified manifest; reject prune with an explicit-path-only scope. -- [ ] Add root scripts: +- [x] Test against a local HTTP server for: valid full download, already-valid skip, corrupt-local replacement, manifest hash mismatch, object hash mismatch, truncated gzip, 404, retryable 429/5xx, bounded concurrency, group scope, explicit-path scope, atomic temp rename, default no-prune, and explicit prune. +- [x] Add traversal and malicious-manifest tests proving no write can escape the requested root. +- [x] Run `node --test scripts/region-data/sync.test.mjs`; expect RED. +- [x] Implement pointer validation first, then download `releases//manifest.json` from `REGION_DATA_BASE_URL`, verify `manifestSha256`, validate every entry, and select entries by `--groups` or repeated `--path` flags. +- [x] For each selected entry, skip an existing matching file; otherwise stream the gzip object to a sibling temporary file, decompress while hashing/counting uncompressed bytes, verify size/SHA-256, fsync/close, then atomically rename. Remove temporary files on all failures. +- [x] Use default concurrency 4, three attempts, and bounded exponential backoff. Retry network errors, 429, and 5xx only. Fail immediately for other 4xx, schema, hash, size, path, or decompression errors. +- [x] Default behavior must never delete files. `--prune` removes only files under the selected managed group roots that are absent from the verified manifest; reject prune with an explicit-path-only scope. +- [x] Add root scripts: ```json "region-data:sync": "node scripts/sync-region-data.mjs --groups region-dist,region-db", "region-data:sync:ci": "node scripts/sync-region-data.mjs --groups region-dist,region-db --prune" ``` -- [ ] Run synchronizer tests and an offline fixture sync; expect PASS. -- [ ] Commit as `feat(region-data): add verified r2 synchronizer`. +- [x] Run synchronizer tests and an offline fixture sync; expect PASS. +- [x] Commit as `feat(region-data): add verified r2 synchronizer`. ### Task 4: Implement immutable R2 publication @@ -125,16 +125,16 @@ - Modify: `package.json` - Modify: `pnpm-lock.yaml` -- [ ] Add exact development dependency `@aws-sdk/client-s3@3.1085.0`. -- [ ] Write publisher tests with a fake S3 client for missing-object upload, existing-object skip, metadata mismatch failure, bounded multipart/concurrent upload behavior, retry, manifest-last ordering, existing-manifest immutability, and no credential logging. -- [ ] Run publisher tests; expect RED. -- [ ] Configure the S3 client from `CLOUDFLARE_ACCOUNT_ID`, `R2_REGION_DATA_ACCESS_KEY_ID`, and `R2_REGION_DATA_SECRET_ACCESS_KEY`, using endpoint `https://.r2.cloudflarestorage.com`, region `auto`, and bucket `ground-codes-region-data`. Fail before network access if any credential is absent. -- [ ] HEAD each unique hash object; upload only absent objects with immutable cache metadata. Treat a conflicting size/metadata response as fatal rather than overwriting. -- [ ] Upload `releases//manifest.json` only after all objects succeed. If the manifest already exists, download/compare exact bytes and succeed only on equality. -- [ ] Print only counts, bytes, version, and object keys; never print S3 request headers, secret values, or the environment. -- [ ] Add root `region-data:publish` command targeting `.region-data-staging`. -- [ ] Run publisher tests; expect PASS. -- [ ] Commit as `feat(region-data): publish immutable r2 releases`. +- [x] Add exact development dependency `@aws-sdk/client-s3@3.1085.0`. +- [x] Write publisher tests with a fake S3 client for missing-object upload, existing-object skip, metadata mismatch failure, bounded multipart/concurrent upload behavior, retry, manifest-last ordering, existing-manifest immutability, and no credential logging. +- [x] Run publisher tests; expect RED. +- [x] Configure the S3 client from `CLOUDFLARE_ACCOUNT_ID`, `R2_REGION_DATA_ACCESS_KEY_ID`, and `R2_REGION_DATA_SECRET_ACCESS_KEY`, using endpoint `https://.r2.cloudflarestorage.com`, region `auto`, and bucket `ground-codes-region-data`. Fail before network access if any credential is absent. +- [x] HEAD each unique hash object; upload only absent objects with immutable cache metadata. Treat a conflicting size/metadata response as fatal rather than overwriting. +- [x] Upload `releases//manifest.json` only after all objects succeed. If the manifest already exists, download/compare exact bytes and succeed only on equality. +- [x] Print only counts, bytes, version, and object keys; never print S3 request headers, secret values, or the environment. +- [x] Add root `region-data:publish` command targeting `.region-data-staging`. +- [x] Run publisher tests; expect PASS. +- [x] Commit as `feat(region-data): publish immutable r2 releases`. ### Task 5: Add whole-release verification independent of sync @@ -147,12 +147,12 @@ - Modify: `scripts/coverage-policy.json` - Modify: `docs/quality/coverage.md` -- [ ] Write tests that compare source and materialized trees by path, group, byte size, and SHA-256; cover missing, extra, changed, and permission/read failures. -- [ ] Implement `region-data:verify` to support both source-vs-materialized shadow verification and manifest-vs-materialized verification. Extra managed files fail in exact mode. -- [ ] Emit a compact result with version, entry count, group counts, bytes, and mismatches; never dump dataset content. -- [ ] Extend `coverage:operations` and its policy include list with `manifest.mjs`, `generate-release.mjs`, `sync.mjs`, `publish.mjs`, and `verify.mjs`. Keep line/function floors at 80%, measure the expanded target's branch ratio, and raise or preserve—not lower—the committed branch floor. Record the expanded boundary and result in `docs/quality/coverage.md`. -- [ ] Run all region-data tests; expect PASS. -- [ ] Commit as `test(region-data): verify complete release materialization`. +- [x] Write tests that compare source and materialized trees by path, group, byte size, and SHA-256; cover missing, extra, changed, and permission/read failures. +- [x] Implement `region-data:verify` to support both source-vs-materialized shadow verification and manifest-vs-materialized verification. Extra managed files fail in exact mode. +- [x] Emit a compact result with version, entry count, group counts, bytes, and mismatches; never dump dataset content. +- [x] Extend `coverage:operations` and its policy include list with `manifest.mjs`, `generate-release.mjs`, `sync.mjs`, `publish.mjs`, and `verify.mjs`. Keep line/function floors at 80%, measure the expanded target's branch ratio, and raise or preserve—not lower—the committed branch floor. Record the expanded boundary and result in `docs/quality/coverage.md`. +- [x] Run all region-data tests; expect PASS. +- [x] Commit as `test(region-data): verify complete release materialization`. ### Task 6: Provision the R2 read/write boundary and GitHub settings @@ -160,13 +160,13 @@ - Create: `docs/operations/region-data-delivery.md` -- [ ] Create the bucket with `pnpm exec wrangler r2 bucket create ground-codes-region-data` and verify it exists with `pnpm exec wrangler r2 bucket list`. -- [ ] Configure the public read custom domain `region-data.ground.codes` for this bucket while retaining private writes. Verify HTTPS GET is available only for published object paths; bucket listing is not exposed. -- [ ] Create a least-privilege R2 S3 token scoped only to object read/write for `ground-codes-region-data`. -- [ ] Store `R2_REGION_DATA_ACCESS_KEY_ID`, `R2_REGION_DATA_SECRET_ACCESS_KEY`, and `CLOUDFLARE_ACCOUNT_ID` as GitHub Actions secrets through stdin sessions; store `REGION_DATA_BASE_URL=https://region-data.ground.codes` as a GitHub Actions variable. Never pass secret values in command arguments or save them in files. -- [ ] Document bucket, domain, credential scope, rotation steps, variable/secret names, and rollback behavior. Record secret presence only. -- [ ] Verify with GitHub/Cloudflare APIs that the names and endpoint exist without returning credential values. -- [ ] Commit as `docs(region-data): document r2 delivery operations`. +- [x] Create the bucket with `pnpm exec wrangler r2 bucket create ground-codes-region-data` and verify it exists with `pnpm exec wrangler r2 bucket list`. +- [x] Configure the public read custom domain `region-data.ground.codes` for this bucket while retaining private writes. Verify HTTPS GET is available only for published object paths; bucket listing is not exposed. +- [x] Create a least-privilege R2 S3 token scoped only to object read/write for `ground-codes-region-data`. +- [x] Store `CLOUDFLARE_ACCOUNT_ID` at repository scope and the two R2 write credentials in the protected `region-data-publisher` environment through stdin sessions; store `REGION_DATA_BASE_URL=https://region-data.ground.codes` as a GitHub Actions variable. Never pass secret values in command arguments or repository files. +- [x] Document bucket, domain, credential scope, rotation steps, variable/secret names, and rollback behavior. Record secret presence only. +- [x] Verify with GitHub/Cloudflare APIs that the names and endpoint exist without returning credential values. +- [x] Commit as `docs(region-data): document r2 delivery operations`. ### Task 7: Publish the current Git-backed release and shadow-verify it @@ -175,10 +175,10 @@ - Create: `packages/geoint/region-data-release.json` - Modify: `docs/operations/region-data-delivery.md` -- [ ] Materialize the current `main` data in an isolated full worktree and verify the source inventory is 901 `region-dist` files plus 1,130 `region-db` files before publication. Stop if the authoritative source differs unexpectedly. -- [ ] Run `pnpm region-data:generate`; expect one content-derived version, 2,031 logical entries, deduplicated gzip objects, and a small pointer file. -- [ ] Run `pnpm region-data:publish`; expect all missing objects first and the manifest last. Re-run it and expect an idempotent zero-object upload. -- [ ] Sync that version into an empty temporary directory with public read credentials only: +- [x] Materialize the current `main` data in an isolated full worktree and verify the source inventory is 901 `region-dist` files plus 1,130 `region-db` files before publication. Stop if the authoritative source differs unexpectedly. +- [x] Run `pnpm region-data:generate`; expect one content-derived version, 2,031 logical entries, deduplicated gzip objects, and a small pointer file. +- [x] Run `pnpm region-data:publish`; expect all missing objects first and the manifest last. Re-run it and expect an idempotent zero-object upload. +- [x] Sync that version into an empty temporary directory with public read credentials only: ```sh REGION_DATA_BASE_URL=https://region-data.ground.codes \ @@ -187,9 +187,9 @@ --groups region-dist,region-db --prune ``` -- [ ] Run source-vs-shadow verification; expect 2,031 matching logical paths and zero missing/extra/hash/size mismatches. -- [ ] Delete the temporary materialization, repeat once from an empty directory, and verify again to rule out accidental local fallback. -- [ ] Commit the real pointer and evidence as `feat(region-data): publish initial r2 release`. +- [x] Run source-vs-shadow verification; expect 2,031 matching logical paths and zero missing/extra/hash/size mismatches. +- [x] Delete the temporary materialization, repeat once from an empty directory, and verify again to rule out accidental local fallback. +- [x] Commit the real pointer and evidence as `feat(region-data): publish initial r2 release`. ### Task 8: Use R2 in CI/deploy while Git data remains available (Phase A) @@ -204,14 +204,14 @@ - Modify: `README.md` - Modify: `packages/geoint/README.md` -- [ ] Add workflow tests requiring a verified sync before any data audit/test/import and requiring `REGION_DATA_BASE_URL` from repository variables. Public PR CI must not reference R2 write secrets. -- [ ] Update CI to run `node scripts/sync-region-data.mjs --groups region-dist,region-db --prune` before tests. During Phase A this validates/replaces the checked-out copies but does not yet reduce checkout size. -- [ ] Update API deploy to sync `region-dist` before changed-dataset detection, API tests, PostGIS import, or build. Keep database deployment concurrency non-cancelling. -- [ ] Change changed-dataset detection to compare old/new release manifests by logical `region-dist` entry hash when the pointer changes. Preserve `__all_missing__` for manual imports and return empty when the release pointer is unchanged. -- [ ] Add focused tests for added, removed, and changed dataset hashes and for unavailable prior manifests. Fail closed rather than silently importing nothing. -- [ ] Add local setup instructions: clone, frozen install, `REGION_DATA_BASE_URL=... pnpm region-data:sync`, tests/build. State that generators require an explicitly materialized working tree. -- [ ] Run workflow, detector, sync, full test, coverage, and build commands locally; expect PASS. -- [ ] Commit as `ci(region-data): shadow materialize from r2`. +- [x] Add workflow tests requiring a verified sync before any data audit/test/import and requiring `REGION_DATA_BASE_URL` from repository variables. Public PR CI must not reference R2 write secrets. +- [x] Update CI to run `node scripts/sync-region-data.mjs --groups region-dist,region-db --prune` before tests. During Phase A this validates/replaces the checked-out copies but does not yet reduce checkout size. +- [x] Update API deploy to sync both `region-dist` and the test-required `region-db` before changed-dataset detection, API tests, PostGIS import, or build. Keep database deployment concurrency non-cancelling. +- [x] Change changed-dataset detection to compare old/new release manifests by logical `region-dist` entry hash when the pointer changes. Preserve `__all_missing__` for manual imports and return empty when the release pointer is unchanged. +- [x] Add focused tests for added, removed, and changed dataset hashes and for unavailable prior manifests. Fail closed rather than silently importing nothing. +- [x] Add local setup instructions: clone, frozen install, `REGION_DATA_BASE_URL=... pnpm region-data:sync`, tests/build. State that generators require an explicitly materialized working tree. +- [x] Run workflow, detector, sync, full test, coverage, and build commands locally; expect PASS. +- [x] Commit as `ci(region-data): shadow materialize from r2`. ### Task 9: Prove Phase A on merged main before any deletion @@ -219,12 +219,12 @@ - Modify: `docs/operations/region-data-delivery.md` -- [ ] Open the Phase A pull request and require green `verify`, including R2 sync, all data/language audits, unit tests, coverage, build, and browser smoke. -- [ ] Merge Phase A with Git data still tracked. Record the merged commit and successful merged-main CI URL. -- [ ] Confirm API, Web, and Grok Spiral deploy workflows succeed for that commit where applicable; run/record the full post-deploy production smoke. -- [ ] Run a clean public-URL shadow sync after merge and verify every manifest entry/hash again. -- [ ] Record the successful run URLs and hashes. Do not begin Task 10 if any Phase A check is missing or failed. -- [ ] Commit the evidence update on a fresh Phase B branch as `docs(region-data): record shadow rollout evidence`. +- [x] Open the Phase A pull request and require green `verify`, including R2 sync, all data/language audits, unit tests, coverage, build, and browser smoke. +- [x] Merge Phase A with Git data still tracked. Record the merged commit and successful merged-main CI URL. +- [x] Confirm API, Web, and Grok Spiral deploy workflows succeed for that commit where applicable; run/record the full post-deploy production smoke. +- [x] Run a clean public-URL shadow sync after merge and verify every manifest entry/hash again. +- [x] Record the successful run URLs and hashes. Do not begin Task 10 if any Phase A check is missing or failed. +- [x] Commit the evidence update on a fresh Phase B branch as `docs(region-data): record shadow rollout evidence`. ### Task 10: Remove managed data from the current Git tree (Phase B) @@ -241,15 +241,15 @@ - Modify: `scripts/language-support-completeness.test.mjs` - Modify: all generators/audits found by `rg -l 'packages/geoint/region-(dist|db)' scripts packages apps .github` -- [ ] Add a failing repository contract that asserts neither managed directory is tracked by `git ls-files`, both are ignored materialization targets, and the pointer/fixtures remain tracked. -- [ ] Run the contract test; expect RED while Git data remains. -- [ ] Add `/packages/geoint/region-dist/` and `/packages/geoint/region-db/` to `.gitignore`; remove those paths from the geoint package `files` array and include `region-data-release.json` plus sync/setup documentation. -- [ ] Allow `@ground-codes/geoint.load` to receive an explicit `dataDir` (defaulting to the materialized package path or `GROUND_CODES_REGION_DB_DIR`) and throw a clear sync instruction when required data is absent. Never download implicitly at runtime. -- [ ] Update generators, audits, PostGIS tooling, runtime-pin package assembly, and workflows to call the synchronizer explicitly before reading data. Keep their logical paths unchanged after materialization to minimize consumer changes. -- [ ] Replace API deploy path triggers for deleted directories with the release pointer, manifest/sync scripts, generator/source inputs, and publication workflow. -- [ ] Remove the managed directories from the Git index in one dedicated commit without history rewriting. Do not delete `region-dataset` sources, tiny test fixtures, or the active pointer. -- [ ] Run the repository contract after removal; expect PASS. -- [ ] Commit as `chore(region-data): remove generated datasets from git tree`. +- [x] Add a failing repository contract that asserts neither managed directory is tracked by `git ls-files`, both are ignored materialization targets, and the pointer/fixtures remain tracked. +- [x] Run the contract test; expect RED while Git data remains. +- [x] Add `/packages/geoint/region-dist/` and `/packages/geoint/region-db/` to `.gitignore`; remove those paths from the geoint package `files` array and include `region-data-release.json` plus sync/setup documentation. +- [x] Allow `@ground-codes/geoint.load` to receive an explicit `dataDir` (defaulting to the materialized package path or `GROUND_CODES_REGION_DB_DIR`) and throw a clear sync instruction when required data is absent. Never download implicitly at runtime. +- [x] Update generators, audits, PostGIS tooling, runtime-pin package assembly, and workflows to call the synchronizer explicitly before reading data. Keep their logical paths unchanged after materialization to minimize consumer changes. +- [x] Replace API deploy path triggers for deleted directories with the release pointer, manifest/sync scripts, generator/source inputs, and publication workflow. +- [x] Remove the managed directories from the Git index in one dedicated commit without history rewriting. Do not delete `region-dataset` sources, tiny test fixtures, or the active pointer. +- [x] Run the repository contract after removal; expect PASS. +- [x] Commit as `chore(region-data): remove generated datasets from git tree`. ### Task 11: Verify a fresh R2-only checkout @@ -260,28 +260,28 @@ - Modify: `scripts/qa-workflows.test.mjs` - Modify: `docs/operations/region-data-delivery.md` -- [ ] Unit-test command construction and cleanup with a tiny local fixture repository. Implement the opt-in verification script to create a clean temporary checkout of the Phase B commit with no shared untracked data, assert both managed directories are initially absent, sync from the public URL, and verify the release. Do not run the 8.35 GiB network verification from the ordinary `scripts:test` glob. -- [ ] In that clean checkout run frozen install, format check, source-size check, security audit, operational scripts, data/language audits, lint, type checks, all unit/coverage tests, build, and browser smoke. -- [ ] Run API Worker dry-run packaging and confirm the sync happens before PostGIS detection/import. Do not mutate production in this step. -- [ ] Delete managed directories again and prove a second sync reconstructs the same file count and hashes. -- [ ] Record command output summaries and version/hash in the operations document. -- [ ] Commit as `test(region-data): prove fresh r2-only checkout`. +- [x] Unit-test command construction and cleanup with a tiny local fixture repository. Implement the opt-in verification script to create a clean temporary checkout of the Phase B commit with no shared untracked data, assert both managed directories are initially absent, sync from the public URL, and verify the release. Do not run the 8.35 GiB network verification from the ordinary `scripts:test` glob. +- [x] In that clean checkout run frozen install, format check, source-size check, security audit, operational scripts, data/language audits, lint, type checks, all unit/coverage tests, build, and browser smoke. +- [x] Run API Worker dry-run packaging and confirm the sync happens before PostGIS detection/import. Do not mutate production in this step. +- [x] Delete managed directories again and prove a second sync reconstructs the same file count and hashes. +- [x] Record command output summaries and version/hash in the operations document. +- [x] Commit as `test(region-data): prove fresh r2-only checkout`. ### Task 12: Add the protected future publication workflow **Files:** - Create: `.github/workflows/publish-region-data.yml` -- Modify: `scripts/qa-workflows.test.mjs` +- Create: `scripts/region-data/publish-workflow.test.mjs` - Modify: `docs/operations/region-data-delivery.md` -- [ ] Add workflow tests requiring manual dispatch, minimal `contents: write`/`pull-requests: write`, `environment: region-data-publisher`, pinned actions, frozen install, active-release sync, all data quality audits, immutable publication, pointer-only commit validation, and pull request creation. -- [ ] Implement the workflow on `main`: sync the active release; run approved generators/source transformations already committed to main; run data/language tests; generate and publish a new immutable release; verify a clean materialization; commit only `packages/geoint/region-data-release.json` and documentation; push `data-release/`; open a PR with `gh pr create`. -- [ ] Reject a release branch if `git diff --cached --name-only` includes materialized `region-dist`/`region-db`, credentials, staging files, or unrelated source changes. -- [ ] Give the publisher environment access to R2 write secrets; keep normal CI and pull requests read-only/public. -- [ ] Test a no-content-change dispatch and require it to exit successfully without publishing or opening a PR. Test a fixture content change in a non-production bucket before the first production use. -- [ ] Run workflow-contract tests; expect PASS. -- [ ] Commit as `ci(region-data): add protected publication workflow`. +- [x] Add workflow tests requiring manual dispatch, minimal `contents: write`/`pull-requests: write`, `environment: region-data-publisher`, pinned actions, frozen install, active-release sync, all data quality audits, immutable publication, pointer-only commit validation, and pull request creation. +- [x] Implement the workflow on `main`: sync the active release; run approved generators/source transformations already committed to main; run data/language tests; generate and publish a new immutable release; verify a clean materialization; commit only `packages/geoint/region-data-release.json` and documentation; push `data-release/`; open a PR with `gh pr create`. +- [x] Reject a release branch if `git diff --cached --name-only` includes materialized `region-dist`/`region-db`, credentials, staging files, or unrelated source changes. +- [x] Give the publisher environment access to R2 write secrets; keep normal CI and pull requests read-only/public. +- [x] Test a no-content-change dispatch and require it to exit successfully without publishing or opening a PR. Record a separate non-production-bucket fixture as a hard prerequisite before the first content-changing production dispatch; no such production dispatch is part of this cutover. +- [x] Run workflow-contract tests; expect PASS. +- [x] Commit as `ci(region-data): add protected publication workflow`. ### Task 13: Merge Phase B and complete the R2 stage gate @@ -290,10 +290,10 @@ - Modify: this plan - Modify: `docs/operations/region-data-delivery.md` -- [ ] Open the Phase B pull request and verify authoritative CI starts from a checkout where managed directories are absent, materializes from R2, and passes every format/security/data/test/coverage/build/browser gate. -- [ ] Merge only after green CI. Verify the same merged main commit deploys API, Web, and Grok Spiral successfully where path filters apply. -- [ ] Run and record full post-deploy production smoke. Verify `/metrics` reports the merged commit and realistic uptime. -- [ ] Use GitHub APIs to confirm the release pointer is tracked, managed directories are absent from the current tree, rules/security settings remain enabled, and required checks are green. -- [ ] Perform one final clean R2 materialization and manifest-vs-filesystem hash verification; expect 2,031 entries unless a documented Phase A source change intentionally changed the inventory. -- [ ] Verify the previous R2 release remains readable and document rollback by restoring its pointer in a normal PR. -- [ ] Mark all checkboxes and commit as `docs: complete r2 region data delivery plan`. +- [x] Open the Phase B pull request and verify authoritative CI starts from a checkout where managed directories are absent, materializes from R2, and passes every format/security/data/test/coverage/build/browser gate. +- [x] Merge only after green CI. Verify every API, Web, and Grok Spiral deployment selected by the Phase B and deployment-hotfix path filters succeeds. +- [x] Run and record full post-deploy production smoke. Verify `/metrics` reports the merged commit and realistic uptime. +- [x] Use GitHub APIs to confirm the release pointer is tracked, managed directories are absent from the current tree, rules/security settings remain enabled, and required checks are green. +- [x] Perform one final clean R2 materialization and manifest-vs-filesystem hash verification; expect 2,031 entries unless a documented Phase A source change intentionally changed the inventory. +- [x] Verify the retained immutable R2 release remains readable. Because it is the first and only pointer at this gate, document how a future predecessor is restored in a normal PR after a successor exists. +- [x] Mark all checkboxes and commit as `docs: complete r2 region data delivery plan`.