Closes #134215

What Problem This Solves

openclaw backup create hangs forever with zero output while the Gateway is running, produces no
archive, and leaves an orphan sqlite-readonly-location.worker process behind. openclaw backup git create --all fails the same way. The documented first-class recovery command is unusable on a live
install — a silent, indefinite non-outcome on a default path.

The cause is not a lock the snapshot waits on. It is a livelock in the stepped online backup.
node:sqlite restarts a stepped backup whenever another connection commits, so a source under
continuous write never converges. Measured on current main, against a live database with one
concurrent writer process:

t+15s steps=8264  restarts=8263  remaining 2262887/2262951 pages
t+30s steps=13305 restarts=13303 remaining 2995670/2995734 pages
t+45s steps=15656 restarts=15654 remaining 3751077/3751141 pages
t+60s steps=17406 restarts=17404 remaining 4467186/4467250 pages
VERDICT: backup never completed - steps=17406, restarts=17404, min remaining=1558759/4467250

17 406 steps, 17 404 of them restarts, and remainingPages never falling below its starting value.
(This probe passed rate: 64 for finer-grained samples; production uses Node's default of 100 pages
per step, so step counts scale accordingly — the page counts and the frozen minimum do not.)
sqlite-snapshot.ts:657 already named the hazard — "Node restarts stepped backups on concurrent
writes" — but nothing bounded it, and nothing above it was bounded either:

• sqlite-snapshot.ts:662 and sqlite-readonly-location.ts:445 — sqlite.backup with no bound.
• sqlite-readonly-location.ts:661 — execFile launching the snapshot worker with only
  { encoding: "utf8" }: no timeout, no killSignal. Its callback fires on child exit, so a
  child stuck inside its own backup strands the parent forever. That is the orphan process in the report.
• sqlite-readonly-location.ts:686 — spawnSync, same omission.
• MAX_SNAPSHOT_ATTEMPTS (:19, loops at :483/:563) bounds the retry count only, so one
  unbounded attempt makes the whole loop unbounded.

Why This Change Was Made

The obvious repair — a timeout on the backup — is the wrong one here, and this repo has already
learned that: #122213 and #122235 replaced timeouts with no-progress detection for the archive
stream after legitimately slow work was being false-timeouted, and that watchdog still lives at
backup-create-stream.ts:92-121. A 13.7 MB snapshot is legitimately slow; a livelocked one is not.
node:sqlite's progress callback distinguishes them exactly: a step that copies pages lowers
remainingPages, and a restart does not.

So the bound is on net page progress, not wall clock:

1. The duplicated "consistent online copy of a live source" policy — open read-only, busy_timeout
   and trusted_schema pragmas, BEGIN, validate, back up, ROLLBACK, close — was copied between
   sqlite-snapshot.ts:648-668 and sqlite-readonly-location.ts:441-451. It now lives in one owner,
   src/infra/sqlite-online-backup.ts, which takes the caller's in-transaction validation as a hook so
   the snapshot path keeps running schema_version, the vec extension load, the integrity check, and
   options.validate inside the same pinned read transaction, in the same order. The consolidation
   also fixes a latent bug: createOnlineReadOnlyBackup previously ran ROLLBACK only on the success
   path, so a mid-backup throw left the read transaction open until close.

2. That owner bounds the copy by elapsed time since it last reached a new all-time-low remaining
   page count, read from the monotonic clock (performance.now(), the repo's own precedent in
   src/cli/daemon-cli/restart-health.ts:385) at the top of the progress callback, plus an unchanged
   plateau guard for a stream that copies nothing at all. Deliberately not a setTimeout: the
   progress callback is the only place production regains control of an in-flight node:sqlite
   backup, so a timer could at best set a flag the next callback reads, and it would make the bound
   untestable without real elapsed time.

   Getting here took three attempts, and the discarded ones are worth stating because they are the
   obvious designs. Counting steps against an all-time minimum is wrong: after a restart Node
   replays remainingPages from the top, so recovery costs pagesCopied / rate steps that look like
   non-progress while the copy runs at full speed — measured at exactly pagesCopied ÷ 100. Counting
   restarts, or wasted work, is also wrong, and that one is only visible with instrumentation:
   both quantities overlap between backups that finish and backups that never do, and both grow with
   database size, so neither can be calibrated. See the table under Evidence.

3. The failure names the cause and a way forward instead of dead-ending: the source is being written
   concurrently, typically by a running Gateway, and quiescing those writes lets the same snapshot
   complete. The remediation text deliberately does not point at openclaw backup sqlite create,
   because that command reaches the same owner (commands/backup-sqlite.ts:85 ->
   snapshot/local-repository.ts:287 -> createOpenClawSnapshotCopy ->
   createVerifiedSqliteSnapshot) and would hit the same bound.

4. The worker launches gain no deadline of their own. An earlier revision added a 30-minute
   timeout + killSignal there; review was right that a fixed cap over the whole worker
   regardless of progress is the defect fix(backup): prevent false archive stalls during active traversal #122213 and fix(backup): keep large file reads from false-timeout #122235 removed, and that mapping its expiry to
   "concurrent writes" asserts a cause it never established. With the backup inside the child bounded
   and rearmed by observed progress, the child is already guaranteed to exit.

No new configuration surface, no schema change, no change to MAX_SNAPSHOT_ATTEMPTS semantics, the
#130962 audit-lease flow, restore/archive formats, or the #127855 child-process lock isolation.

User Impact

openclaw backup create and openclaw backup git create now terminate instead of hanging forever.
Under continuous writes they either complete — measured above, 66 s with a live writer — or fail with
an actionable message naming the cause, rather than producing nothing and stranding an orphan worker.
When they can copy the database they behave exactly as before, including on large, slow snapshots.
See the open question below for the one case where the current threshold could still cut a converging
copy short.

Evidence

All measurements are real runs, not estimates.

Reproduction of the defect on current main (d2a616bdf373), Linux 6.8.0-138 x86_64, Node
v24.20.0, SQLite 3.53.4, inside a node:24 Docker container. The exact production chain
(createVerifiedSqliteSnapshot -> withSqliteSnapshotSource -> prepareSqliteReadOnlyLocation) was
driven against a live database that a separate writer process kept committing to:

Reported honestly as a race: the same scenario hung once and completed on a rerun. That is why the
defect is intermittent for operators and hard to reproduce, and it is why the fix is a bound rather
than a change to the copy strategy. The instrumented run quoted above pins the mechanism, and the
control — the identical snapshot with the writer stopped — completes in 733.8 s on the grown database.

Real openclaw backup create on this PR head, built from the branch (pnpm build) and run in a
node:24 container against a 145 MB WAL state database created by openclaw doctor, with a separate
process committing continuously — the reported scenario:

CASE 1: openclaw backup create WITH a concurrent writer
  exit=0 after 66s
  Backup archive: /tmp/out/2026-09-03T22-19-44.884+00-00-openclaw-backup.tar.gz
  Created ... (4,512,378 bytes)
  orphan snapshot workers: (none)

CASE 2: same command, writes quiesced
  exit=0 after 456s
  Created ... (64,869,733 bytes)

On the reported version this command hung with zero output past 180 s and left an orphan
sqlite-readonly-location.worker. On this head it finishes while the writer is still committing,
produces a real archive, and leaves no orphan process.

Two honest notes. First, case 1 completed rather than tripping the new bound, so this run proves the
hang is gone and the healthy path works under contention — it does not exercise the bounded-failure
message end-to-end through the CLI; that path is covered by the unit and owner-boundary tests below,
because forcing it through the CLI means driving the 30-minute deadline. Second, case 1 printed
[sqlite/transaction] SQLite transaction lock wait failed and Warning: the backup outcome could not be recorded: database is locked — the archive was still produced; that is a separate contended-write
path recording the outcome, not the snapshot, and this PR does not change it.

Calibration. Twelve instrumented runs against real WAL sources with a real node:sqlite backup
and an in-process writer at a fixed cadence:

Read the last three columns across the two groups. Restart counts overlap (completing runs needed
up to 38; a stalled one sat at 406). Discarded work overlaps far worse — a run that finished in
18.7 s had thrown away 107 full copies, more than two of the stalled runs. Both quantities also grow
with database size, so a constant calibrated at 63 K pages means nothing at 4,467,250. The last column
does not overlap at all: every completing run reached a new low within 38–1,076 ms, every stalled
run sat 120–320 seconds without one. That is the axis the bound uses.

Two-sided proof, Linux Docker. With both bounds set to POSITIVE_INFINITY — the pre-fix
unbounded behaviour — exactly the four non-convergence cases fail and every anti-false-abort case
stays green:

BASE (MAX_MS_WITHOUT_NET_PROGRESS = MAX_STEPS_WITHOUT_COPY_ADVANCE = Infinity)
  Tests  4 failed | 5 passed (9)

HEAD (8 suites)
  Test Files  8 passed (8)
  Tests  223 passed | 2 skipped (225)

The suite covers both directions: shapes that must abort (restart on nearly every step, restart every
few steps without beating the best, a stream that never moves, and headway so slow it costs a whole
pass per page) and shapes that must not (a long strictly-decreasing copy, a single restart followed by
recovery far longer than the bounds, and many restarts each reaching a new all-time low). One case
drives a real, unstubbed node:sqlite backup to confirm that throwing from progress rejects it.

Backward clock correction (added after review). The deadline is read from performance.now(),
so a wall-clock step cannot postpone it. The added test arms the deadline, sets the system clock back
ten minutes, then feeds one-minute livelock cycles, and pins the abort at cycle 31 (thirty one-minute
cycles exhaust the budget). On the previous head, which read Date.now(), the same test fails with the
deadline postponed by exactly the rollback:

previous owner (Date.now):    sqlite-online-backup.test.ts  Tests  1 failed | 9 passed (10)
                              AssertionError: expected 41 to be 31
this head (performance.now):  sqlite-online-backup.test.ts  Tests  10 passed (10)
Linux Docker node:24 (VPS), same two legs:
  previous owner:   sqlite-online-backup.test.ts  Tests  1 failed | 9 passed (10)   expected 41 to be 31
  this head:        4 suites (online-backup, backup-sqlite-snapshot, readonly-location, sqlite-snapshot)
                    Tests  92 passed | 2 skipped (94)

node:sqlite progress contract, rechecked upstream. Node v26.x reference for
sqlite.backup(sourceDb, path[, options])
(added in v23.8.0 / v22.16.0), quoted:

rate <integer> Positive number of pages to be transmitted in each batch of the backup. Default: 100.
progress <Function> An optional callback function that will be called after each backup step. The
argument passed to this callback is an <Object> with remainingPages and totalPages properties,
describing the current progress of the backup operation.
Returns: <Promise> A promise that fulfills with the total number of backed-up pages upon completion,
or rejects if an error occurs.
The backed-up database can be used normally during the backup process. Mutations coming from the same
connection - same <DatabaseSync> - object will be reflected in the backup right away. However,
mutations from other connections will cause the backup process to restart.

And the same contract exercised against the real API, no OpenClaw code involved (Node v26.5.0,
SQLite 3.53.3, 16-page WAL source):

progress payload keys = ["totalPages","remainingPages"]
progress sequence (rate: 4): [{"totalPages":16,"remainingPages":12},{"totalPages":16,"remainingPages":8},{"totalPages":16,"remainingPages":4}]
throw from progress -> rejected with the thrown error | progress calls before reject = 1
remainingPages with one foreign commit after step 2 (rate: 2): [14,12,15,13,11,9,7,5,3,1]

The last line is the restart the docs describe: a commit from another connection sends remainingPages
back up (12 → 15), which is exactly the signal the bound keys on. The docs do not promise that a throw
from progress rejects the backup; that is observed behaviour, and the unit test rejects when a real node:sqlite progress callback throws pins it on every CI run (Node 24) so a change in Node would show
up there rather than in production.

Data-model compatibility. There is no DDL in this diff, no schema-version bump, no new table, column,
or index; the review tool's "SQLite table change" flag matched the PRAGMA schema_version and
PRAGMA trusted_schema reads moving from sqlite-snapshot.ts / sqlite-readonly-location.ts into
the owner, unchanged in meaning. Copy fidelity, measured through backupSqliteOnline on a WAL source
with two STRICT tables, an index, a view, a trigger, user_version = 41 and 500 rows:

sqlite_master rows in source: 7
copy sqlite_master == source sqlite_master: true
copy pragmas: user_version 41, page_size 4096, integrity_check ok, sessions 500
source untouched by backup: true

The copy's schema_version cookie is written by SQLite's own backup API (sqlite3_backup_finish sets
the destination's previous cookie + 1) and the same sqlite.backup call did that on main, so nothing
about how a copy is read back or upgraded has changed.

Contention classification across the snapshot worker (added after review). A rollback-journal
source is snapshotted inside the isolated read-only worker, which used to serialize only an error
message; the parent rebuilt a plain Error, so a contended copy on that path reached the backup
command without its typed cause and lost the --only-config remedy. The worker now reports a closed
failure code (failure: "contention" | "error") next to the message, the parent validates the closed
shape and rebuilds the typed cause through one small owner factory (the error class stays private),
on both the execFile and spawnSync routes since they share the same adopter. Regression
preserves online-backup contention across the isolated worker boundary drives the public
prepareSqliteReadOnlyLocation in a child process on a rollback-journal source, with a
NODE_OPTIONS --import preload that replaces node:sqlite's backup with a wedged progress stream so
the real owner's plateau guard throws a real contention error inside the real grandchild worker; the
child reports whether isSqliteBackupContentionError holds in the parent:

previous head (message only):  preserves online-backup contention across the isolated worker boundary
                               AssertionError: expected false to be true
this head (typed cause):       4 suites  Tests  92 passed | 3 skipped (95)
Linux Docker node:24 (VPS), same two legs:
  previous head:   sqlite-readonly-location.test.ts  Tests  1 failed | 50 passed (51)   expected false to be true
  this head:       4 suites  Tests  93 passed | 2 skipped (95)

Suites. On Linux (Docker node:24), sqlite-online-backup, sqlite-readonly-location,
sqlite-snapshot, backup-create, backup-create.legacy-audit-lease, backup-archive-publication,
and state-migrations.source-snapshot all pass: 218 passed, 2 skipped, 7 files, including the
existing concurrent-writer coverage at sqlite-readonly-location.test.ts:652, :725, and :768,
unchanged. node scripts/check-changed.mjs passes for every changed file.

One test, backup-create.test.ts > sanitizes every in-state symlink and hardlink alias of a canonical agent SQLite DB, times out at 120 s on macOS. It fails identically on the unmodified base commit on
the same machine, and passes on Linux, so it is pre-existing local slowness rather than a regression
from this change.

Production LOC delta. git diff --numstat against the base:

 91  0  src/infra/backup-sqlite-snapshot.test.ts
  8  1  src/infra/backup-sqlite-snapshot.ts
296  0  src/infra/sqlite-online-backup.test.ts
118  0  src/infra/sqlite-online-backup.ts
115  0  src/infra/sqlite-readonly-location.test.ts
 35 42  src/infra/sqlite-readonly-location.ts
  6  9  src/infra/sqlite-readonly-location.worker.ts
  8 22  src/infra/sqlite-snapshot.ts

Production net +101 (new owner +118, backup boundary +7, sqlite-readonly-location.ts −7,
worker −3, sqlite-snapshot.ts −14); tests +502. Roughly half the owner's lines are the comment recording
what the constant was calibrated against; the rest is the bound, the typed error and its predicate.
Absorbing the duplicated backup policy into the owner is the net-negative half of the diff and is what
keeps the production delta this small.

Open design question for maintainers

The 30 minutes is a product decision, not a measurement, and it is yours to set.

An online backup of a continuously written database has no guaranteed finish time — node:sqlite
discards the whole pass every time another connection commits — so the only real question is how long
an operator should wait before being told to quiesce writes. The measurements above rule out every
alternative axis: restart counts and wasted work both overlap between the populations and grow with
size. Elapsed time since the copy last beat its own best is the one quantity that separates them.

A caveat I want to put in front of you rather than bury. Independent review ran the experiment my
own table lacks — holding contention constant relative to pass duration instead of at a fixed writer
interval — and found the gap without a new low is flat at 0.20–0.24 pass-durations across
20 K/60 K/200 K pages while the absolute figure grows 65 → 138 → 526 ms. Uncontended pass duration is
linear in size. So the size-invariant quantity is the gap measured in pass-durations, and an
absolute millisecond threshold silently re-introduces a size dependence. Real completing backups in
that experiment needed up to 11.6 pass-durations; against the 733.8 s control for the reported
4,467,250-page source, 30 minutes buys only 2.45. My table cannot refute this — its cadence was
raised with size (55 ms at 21 K, 160 ms at 63 K, 2 s at 211 K), so it contains no high-contention
large-database point that completed.

Concretely: at the reported database size this constant could abort a copy that would have
finished. The suggested fix is to derive the budget from observed pass cost inside the callback
(totalPages is already in the payload) and keep 30 minutes as a ceiling rather than the whole rule.
I have not done that here because it changes the policy surface and I would rather you chose it.

The other disclosed cost: the reported livelock now fails after 30 minutes rather than in ~1.2 s.
A smaller number buys a faster failure there and increases the false-abort risk above. If you want the
fast abort back, the honest options are a shorter deadline or a second, explicitly-owned cap — not a
return to counting restarts, which the data refutes.

Known gaps I would fix on request, none of which affect the hang itself: the copy is still silent
while it runs, so the reported "zero output" becomes "silent for up to the deadline, then a good
error" — the repo's own backup-create-stream.ts:92-121 progress reporting is the precedent; the
retry paths (MAX_SNAPSHOT_ATTEMPTS, withSqliteSnapshotSource) each restart the window rather than
sharing one deadline; and on the prepared-copy branch the message names the
staging path rather than the operator's database.

One more result worth your attention on its own: at 211 K pages under a 160 ms writer, nothing
completed at all within 400 s, and at 400 ms it was still at 77 % after 401 s. That suggests a large,
continuously written database may simply have no finishing time for an online backup, and that saying
so promptly is the useful behaviour rather than a failure of this bound.

CI status

Head d82ca04eb9 is the branch rebased onto d5992278b3 (main as of 2026-09-05) so that #137999's
errcode suffix in the snapshot worker message and this PR's failure tag coexist; the five commits are
unchanged in content, and the expectation in the upstream worker diagnostics test (added by #137999) now
includes the failure tag. Earlier checks below were taken on the pre-rebase heads and re-run on this one.

Rebased onto 8c6653a78d to pick up the createBackupResourceInventory signature change
(configPath/oauthDir → configPaths/oauthDirs), which my new test had to follow;
pnpm check:test-types is green.

The remaining red checks — checks-node-compact-small-49, -small-55, -large-2 and the
openclaw/ci-gate that aggregates them — are already red on main and unrelated to this change.
Scoped proof, same machine, same command:

this branch:                 test/scripts/full-release-candidate-reuse.test.ts  Tests  2 failed | 28 passed (30)
unmodified main @ 8c6653a78d: test/scripts/full-release-candidate-reuse.test.ts  Tests  2 failed | 28 passed (30)

Both failures are full release candidate discovery CLI > marks bounded candidate evaluation unavailable ... and ... marks a selected candidate with a missing constituent unavailable, in
release-candidate tooling that this PR does not touch. Per AGENTS.md I have not tried to fix them
here. security-fast also failed on an earlier push and is likewise red on main
(run 33813389736); it is green on this head.

Notes

• clawsweeper:no-new-fix-pr on the issue reads as "does not recommend queueing a new automated
  fix PR", and its review asked for a live reproduction before an automated patch. That reproduction
  is above and was posted to the issue before this PR was opened; this is a human-authored PR, not a
  queued automated one. Happy to stand down if that reading is wrong.
• The design question of whether the bound belongs in one extracted owner or only at the
  createVerifiedSqliteSnapshot boundary is open on the issue; this PR implements the extracted-owner
  version and can be reshaped.
• Maintainer edits are enabled on this branch.

Human + AI disclosure: the reproduction, analysis, implementation, and review were produced by me
working with AI coding agents, across four independent read-only review passes plus a second round
against the settled commit. Two of those passes returned NO-GO on the first implementation and are
the reason the bound was rebuilt and the wall-clock timeout dropped; their measurements are quoted
above. Every number in this PR comes from the runs shown.