Skip to main content
← Back to list
01Issue
BugOpenSwamp CLI
AssigneesNone

Relationships

#1638 Catalog write-lock contention during extension load leaves a partial registry; auto-resolve fabricates 'local edits' and suggests a destructive --force re-pull

Opened by psftw · 8/13/2026

Summary

If another process holds the extension catalog's SQLite write lock for longer than the busy timeout (~5 s) while a swamp invocation needs to (re)index a pulled extension, that extension silently fails to load: the failure is logged once at WARN and execution continues on a partially populated registry. The first lookup of the missing type then triggers the auto-resolve fallback, which fabricates a different cause and prescribes destructive advice:

2026-08-13T12:32:02.520Z [WRN] swamp·cli: Failed to load user model "ssh.ts": "Error: database is locked"
   Resolved   nestpoke (@sbx/nested)
  Executing   poke
  Resolving   @swamp/ssh not found locally, searching registry...
      Error   @swamp/ssh already installed at <repo>/.swamp/pulled-extensions/@swamp/ssh but failed to load
              Local edits may be preventing it from registering — inspect the source and fix errors.
              To reset: swamp extension pull "@swamp/ssh" --force
Error: nested resolve failed: Model type "@swamp/ssh" not found for definition "testfleet".

Note the contradiction: the WARN names the real cause (database is locked), then the ERROR three lines later invents another ("local edits") and recommends swamp extension pull --force — the one command documented to discard user work — for a pristine, current, checksum-verified install that merely lost a lock race. The method run fails with the unrelated "Model type not found"; nothing lands in the per-run log except the step-start INFO line.

Two real swamp invocations racing to re-index (e.g. the first runs after a manifest change) can hold the write lock long enough to do this to each other — that is how it was hit in the wild (two parallel model method run <role> check invocations; the pulled @swamp/ssh type vanished while the repo's local types, loaded earlier in the phase, worked).

Reproduction

Verified against release 20260812.161650.0-sha.7abaf01a (Linux). Repo: pulled @swamp/ssh 2026.07.28.1, one @swamp/ssh definition (testfleet, any host list — resolve never connects), and a minimal local extension model whose method calls ctx.runModel({definition: "testfleet", method: "resolve", ...}) (mirroring how @psftw/pets calls its fleet).

  1. Dirty the index so the load phase must write:

    sqlite3 .swamp/_extension_catalog.db \
  "DELETE FROM bundle_types WHERE type_normalized='@swamp/ssh'"

    (Any state requiring a row write works — a source mtime bump does too. With a warm catalog the load phase is read-only and the race window closes, which is why the bug strikes "first run after a change" and then seems to fix itself: any later uncontended run heals the row.)

  2. Hold the write lock from a second process, as a slow peer would:

    python3 -c "import sqlite3,time; db=sqlite3.connect('.swamp/_extension_catalog.db');
db.execute('BEGIN IMMEDIATE'); time.sleep(30); db.rollback()" &

  3. swamp model method run nestpoke poke → the output above, every time.

Threshold: a ≤2 s hold is absorbed (busy timeout); ≥6 s reproduces the cascade 100%. Deleting the row without the lock self-heals silently, and a corrupt bundle surfaces its real SyntaxError — the misleading path is specifically transient faults during the index build.

FD exhaustion behaves correctly in this path, for contrast: at ulimit -n 24 the run fails with an honest "too many open files". The lock-loss path is the one that lies.

Mechanism

  • loadUserModels (src/cli/mod.ts): per-file failures from loader.buildIndex are warn-and-continue, and the surrounding catch warn-and-continues wholesale. Load order is local > sources > pulled, so a mid-phase fault yields exactly the observed asymmetry: local types registered, pulled types absent.
  • ExtensionAutoResolver.installAndLoad (src/domain/extensions/extension_auto_resolver.ts): on a registry miss with an intact on-disk install it emits renderAutoResolveAlreadyInstalled, whose text guesses "local edits". The renderer receives only (extension, path) — no error object exists at that point, so the printed cause is fabricated by construction.
  • Same defect family as #1620 (I/O faults degrading into "model not found" in definition walks — shipped), #741 (datastore load path; its "secondary defects" section describes this exact message firing on a pristine install), #585, #524.

Fix (implemented and validated)

Patch to loadUserModels (applies to current main; the region is unchanged since the failing release), attached as extension-loader-fix.patch:

  1. Classify transient index-build failures (database is locked / EMFILE / EACCES / EIO / …).
  2. Retry buildIndex up to 2× with backoff — a peer's write transaction is normally sub-second, so the race is absorbed invisibly.
  3. If transient failures persist, throw with the real error instead of continuing on a partial registry: Failed to load 1 extension model(s): ssh.ts: Error: database is locked. If another swamp process is writing the extension index, retry once it finishes.

Validation (same repro recipe):

Scenario Release 20260812 Patched
No contention ok ok
8 s writer fabricated "local edits" cascade 1 retry → succeeds
45 s writer fabricated cascade honest failure naming the lock

A production version should use the established UserError type for clean rendering (the demo throws a plain marker class, so a stack trace prints), apply the same treatment to loadUserVaults / loadUserDatastores / loadUserReports, and could reuse/extend src/infrastructure/persistence/io_errors.ts from #2125 for the classification. Independently worthwhile: when auto-resolve's "already installed but failed to load" branch fires, surface the load failure recorded earlier in the process instead of the "local edits" guess — the information exists in buildIndex's failure list.

Environment

  • Reproduced on release 20260812.161650.0-sha.7abaf01a and from source at c8953e9e (20260811) — behavior identical; not a regression, a longstanding path.
  • Linux 6.12.101+deb13-amd64; @swamp/ssh 2026.07.28.1 (pulled, pristine).
02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

8/13/2026, 1:42:29 PM

No activity in this phase yet.

03Sludge Pulse

Sign in to post a ripple.