Skip to content

fix(update): clear source leases from the rehearsal state copy - #140004

Merged
steipete merged 1 commit into
mainfrom
steipete/fix-update-rehearsal-copy-leases
Sep 6, 2026
Merged

steipete merged 1 commit into
mainfrom
steipete/fix-update-rehearsal-copy-leases

Conversation

@steipete

@steipete steipete commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

Related: #139701, #138839

What Problem This Solves

Fixes an issue where users updating a running Gateway would fail candidate migration rehearsal because the copied state still claimed that the serving Gateway owned its agent databases. The live Gateway correctly keeps serving through candidate validation, so its process remains alive and Doctor correctly refuses those copied claims.

Why This Change Was Made

Clear source agent-database and state-coordination leases in the private shared-database snapshot before publishing the rehearsal copy. Candidate processes then acquire their own ownership through the existing maintenance and runtime paths.

This is separate from concern E in the #135868 split because the repair belongs to the candidate-state snapshot owner. This is an implementation repair under the existing ownership-routing carve-out: there is no schema, schema-version, table-definition, migration, stored-representation, retention-rule, or public-contract change. It only clears lease rows in the throwaway rehearsal copy created by the canary; the live store and its lease semantics are unchanged. The material SQLite/persistent-store review checkpoint does not apply to this copy-only repair.

User Impact

Updates can validate a candidate while the previous Gateway remains running. Doctor continues to refuse maintenance against databases actually held by a live process.

Evidence

  • A packaged launchd A→B trial for fix(update): preserve restart outcomes and update-history ownership #139701 reproduced the refusal before activation. The old Gateway stayed healthy on the same process through 200 post-failure samples over 206 seconds. An independent installed snapshot-worker probe confirmed that the copy retained the serving process's two live agent-database leases.

  • The existing DELETE/WAL snapshot cases now initialize the canonical shared schema, seed a live PID/start identity and an unexpired maintenance claim, run the real snapshot worker, and require Doctor maintenance admission on the copy. Both copied lease tables must be empty; source database bytes and directory contents remain unchanged. Registry rebinding, durable data, schema versions, missing tables, and external-path coverage remain.

  • Focused candidate-state tests: 6 passed. Snapshot/canary/lease-owner/heartbeat coverage: 75 passed, 2 existing platform skips, including built-candidate startup/readiness and process-group teardown. Local autoreview found no actionable P0–P2 findings. The full changed-file gate passed twice, including after rebase. Branch autoreview is clean at P0–P2. The lockfile is unchanged.

  • Counterproof from committed checkpoint c535fab8e59: removing only the production repair makes both DELETE/WAL cases fail with the existing maintenance-lease timeout. Restoring the exact reviewed bytes makes both pass. The unchanged live-owner maintenance-refusal regression also passes.

  • Fresh launchd A→B and cold-C trials follow when fix(update): preserve restart outcomes and update-history ownership #139701 is rebased onto this repair, per the maintainer-requested sequence. They are not claimed as completed proof for this PR.

  • Exact-head CI passed on 5df208c0cd5d7c6d24bb76df73faa13628eccceb: run 34026108855. ClawSweeper’s review of that head found no actionable or security findings and listed no before-merge or rank-up requests.

Production growth is limited to enforcing ownership in the private copy; there is no new abstraction or fallback. The broad archive sanitizer is intentionally unused because it also removes unrelated queued data and changes approval information.

Class Added Deleted Net Before → after
Production 12 3 +9 278 → 287
Tests/support 51 9 +42 260 → 302
Tooling 0 0 0 0 → 0
Docs 0 0 0 0 → 0

@clawsweeper

clawsweeper Bot commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@steipete
steipete marked this pull request as ready for review September 6, 2026 09:49
@openclaw-barnacle openclaw-barnacle Bot added size: S maintainer Maintainer-authored PR labels Sep 6, 2026
@steipete
steipete force-pushed the steipete/fix-update-rehearsal-copy-leases branch from cc6e27d to 5df208c Compare September 6, 2026 09:58
@clawsweeper clawsweeper Bot added P1 High-priority user-facing bug, regression, or broken workflow. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Sep 6, 2026
@clawsweeper

clawsweeper Bot commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed September 6, 2026, 6:12 AM ET / 10:12 UTC.

ClawSweeper review

What this changes

Clear process and coordination leases from disposable update-rehearsal database copies so candidate validation can proceed while the serving Gateway remains running.

Merge readiness

✅ Ready for maintainer review

This remains a useful, focused repair: inspected main still copies live ownership claims into rehearsal state. No actionable correctness or security findings remain, and the related update work explicitly treats this PR as a separate prerequisite.

Priority: P1
Reviewed head: 5df208c0cd5d7c6d24bb76df73faa13628eccceb

Review scores

Measure Result What it means
Overall readiness 🐚 platinum hermit (4/6) A narrow owner-level repair with meaningful regression coverage, reported counterproof, and no blocking findings.
Proof confidence 🌊 off-meta tidepool Not applicable: The maintainer-authored PR is exempt from ordinary contributor proof requirements. Reported worker regressions exercise copied-state maintenance admission; fresh packaged update trials are explicitly deferred to the dependent PR and are not counted as completed after-fix proof here.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Not applicable Not applicable: The maintainer-authored PR is exempt from ordinary contributor proof requirements. Reported worker regressions exercise copied-state maintenance admission; fresh packaged update trials are explicitly deferred to the dependent PR and are not counted as completed after-fix proof here.
Evidence reviewed 8 items Current main still needs the repair: The fetched main implementation rebinds registry paths but does not clear either lease table in its snapshot transform.
Introduced change is confined to the private copy: The introduced deletion runs only in the shared-database snapshot transform. The snapshot helper supplies a private staged database, verifies it, and publishes it after transformation; it does not pass the serving database handle to this callback.
Rehearsal owns a separate state directory: The canary allocates a fresh temporary directory, directs state/config/agent paths into it, waits for snapshot completion, and then invokes candidate Doctor. Documentation defines this as isolated rehearsal while the old Gateway serves.
Findings None None.
Security None None.

How this fits together

The updater copies live databases into a private temporary directory before testing a candidate installation. Doctor acquires maintenance ownership of those copies, and successful rehearsal permits the updater to proceed toward activation.

flowchart TD
  A[Serving Gateway databases] --> B[Private database snapshots]
  B --> C[Clear copied ownership claims]
  C --> D[Rebind copied database paths]
  D --> E[Doctor acquires fresh maintenance ownership]
  E --> F[Candidate validation result]
Loading

Before merge

None.

Agent review details

Security

None.

PR surface

Source +9, Tests +42. Total +51 across 2 files.

View PR surface stats
Area Files Added Removed Net
Source 1 12 3 +9
Tests 1 51 9 +42
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 2 63 12 +51

Review metrics

None.

Technical review

Best possible solution:

Keep rehearsal ownership independent of serving-process ownership while preserving live maintenance refusal and the copied data used for candidate validation.

Do we have a high-confidence way to reproduce the issue?

Yes: source shows copied live claims reaching unchanged maintenance refusal, and the supplied discussion records that failure in an installed update trial. This review did not execute a reproduction.

Is this the best way to solve the issue?

Yes: clearing invalid ownership claims at the private-copy producer repairs the boundary without weakening Doctor or applying unrelated archive sanitization.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against d7217198496e.

Labels

Label justifications:

  • P1: A packaged update trial exposed a current-main rehearsal failure while a healthy Gateway retains its legitimate live leases.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The maintainer-authored PR is exempt from ordinary contributor proof requirements. Reported worker regressions exercise copied-state maintenance admission; fresh packaged update trials are explicitly deferred to the dependent PR and are not counted as completed after-fix proof here.

Evidence

What I checked:

  • Current main still needs the repair: The fetched main implementation rebinds registry paths but does not clear either lease table in its snapshot transform. (src/infra/update-candidate-state.ts:236, d7217198496e)
  • Introduced change is confined to the private copy: The introduced deletion runs only in the shared-database snapshot transform. The snapshot helper supplies a private staged database, verifies it, and publishes it after transformation; it does not pass the serving database handle to this callback. (src/infra/update-candidate-state.ts:239, 5df208c0cd5d)
  • Rehearsal owns a separate state directory: The canary allocates a fresh temporary directory, directs state/config/agent paths into it, waits for snapshot completion, and then invokes candidate Doctor. Documentation defines this as isolated rehearsal while the old Gateway serves. (src/infra/update-candidate-canary.ts:189, 5df208c0cd5d)
  • Meaningful regression coverage and preserved refusal: Both DELETE/WAL cases seed live process identity and an unexpired maintenance claim, invoke the real snapshot worker, require maintenance admission on the copy, and verify unchanged source bytes and directory entries. Existing lease-owner coverage separately rejects maintenance before its callback when a live owner remains. These tests were inspected, not executed during this read-only review. (src/infra/update-candidate-state.test.ts:65, 5df208c0cd5d)
  • Existing sanitizer has a different contract: The archive sanitizer also removes queued deliveries and expiring blobs and rewrites approval information. Applying that broader transform to candidate rehearsal would change unrelated validation inputs. (src/state/openclaw-state-snapshot-sanitizer.ts:37, 5df208c0cd5d)
  • Merged feature history: History connects the snapshot owner to merged fix(update): validate candidates before stopping the Gateway #138839. Raw commit records identify parent 7e55839; comparison against that parent records the snapshot file as added. The current transform line remains attributed to that merged commit. (src/infra/update-candidate-state.ts:239, d1a69c0b2095)

Likely related people:

  • Peter Steinberger: Raw commit d1a69c0 adds src/infra/update-candidate-state.ts:236 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: d1a69c0b2095; files: src/infra/update-candidate-state.ts)

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (1 earlier review cycle)
  • reviewed 2026-09-06T10:03:15.824Z sha 5df208c :: needs maintainer review before merge. :: none

@steipete
steipete merged commit 6a7ed06 into main Sep 6, 2026
107 checks passed
@steipete
steipete deleted the steipete/fix-update-rehearsal-copy-leases branch September 6, 2026 10:13
@steipete

steipete commented Sep 6, 2026

Copy link
Copy Markdown
Contributor Author

Merged via squash.

github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Sep 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

maintainer Maintainer-authored PR P1 High-priority user-facing bug, regression, or broken workflow. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: S status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant