Skip to main content
← Back to list
01Issue
FeatureClosedSwamp CLI
Assigneesstack72

Relationships

#812 Same-namespace writers fully serialize on the per-namespace lock — could maintenance/append writes avoid holding it?

Opened by mgreten · 6/25/2026

Following up on the question I left at the end of #666 about where to track the remaining intra-namespace piece — filing it here so the repro has a clean home. Happy to fold this back into #666 if you'd rather reopen that instead; #666's lock-scoping work genuinely shipped, so a fresh issue felt cleaner than reopening a closed one. This is also the post-per-namespace successor to #520 (same 60s-LockTimeout cascade symptom, now on the S3 datastore's .locks/<namespace>.lock rather than the old per-model/filesystem lock).

What happens

After the per-namespace lock split shipped (@swamp/s3-datastore@2026.06.24.1, thank you), cross-namespace writes no longer contend — but writers within one namespace still serialize completely, with no queuing or fairness. The loser hard-fails at the 60s maxWaitMs rather than waiting its turn.

Repro — 4 concurrent datastore sync against the same namespace, creds healthy (~20ms), lock clean, nothing else running:

  • run A: Sync complete (1 pulled) — held the lock ~50s
  • run B: LockTimeoutError after 60500ms (same holder)
  • run C: LockTimeoutError after 60746ms (same holder)
  • run D: LockTimeoutError after 60266ms (same holder)

So 1 winner + N-1 timeouts. The winner's ~50s hold is dominated by pulling/pushing the whole ~110 MB .datastore-index.json under the lock. Two ordinary same-namespace calls at once (e.g. a workflow step + an unrelated model method in the same repo) reliably produce a 60s timeout on the loser — not an edge case. A data gc that expired just 7 entries held the lock ~5.5 min for the same reason (full-index rewrite under lock).

The thing I keep wondering

Much of what contends here isn't read-modify-write on shared state — it's append / maintenance writes: auto-generated per-run report artifacts (#811), data gc, high-frequency pollers each writing their own data names. They don't conflict with each other's data; they only collide because every write rewrites the one shared manifest under the lock. So the lock is protecting the manifest, not their data.

That suggests the contention might be avoidable rather than just shortenable. A few directions, non-exclusive — wholly deferring to what fits the architecture:

  1. Decouple the write from the manifest rewrite (a commit queue / append log). Writers append their index delta to a cheap per-namespace log and return; a single background compactor coalesces deltas into the manifest periodically. The pollers/gc/artifact writes never block on the big rewrite — one owner pays the merge cost. This is the one I'm most curious about: it would let these writes proceed without waiting on the lock at all.
  2. Incremental / sharded index (the structural fix noted in #666 and #811): a write touches only its own shard, so lock-hold is proportional to the write, not the whole 110 MB. Shrinks the window rather than removing the lock.
  3. Lock fairness / FIFO queue: even if a hold stays long, a queued caller waits its turn instead of hard-erroring at 60s. The floor-level mitigation — turns a hard failure into latency.

(1) and (2) compose; (3) is independently worth having for the failure mode. Our real-world trigger is several launchd pollers writing one namespace, which the per-namespace split correctly doesn't separate — so the within-namespace path is where it bites. Glad to share more profiling or repro detail.

Environment

02Bog Flow
✓OPEN○TRIAGED○IN PROGRESS◉CLOSED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 4 MOREPR_MERGED

Closed

7/14/2026, 11:19:04 PM

No activity in this phase yet.

03Sludge Pulse
stack72 assigned stack727/1/2026, 3:45:57 PM
Editable. Press Enter to edit.

mgreten commented 6/25/2026, 8:23:15 PM

Adding a consumer-side data point that reinforces the "append/maintenance writes shouldn't contend" framing — observed repeatedly during real multi-process use, not a synthetic repro.

Symptom: a write-on-every-call audit pattern silently degrades under namespace-lock contention

I run an autonomous build/QA pipeline as a long workflow run. Several scheduled pollers and a couple of interactive sessions operate in the same namespace concurrently. Whenever the workflow holds the namespace lock, other model-method calls from the concurrent processes hit the 60s LockTimeoutError you describe — 1 winner + N-1 timeouts, exactly as in your repro.

The part worth flagging: a large share of the contending calls are provider-resolution / decision calls that are read-mostly. The method computes a pure result from its inputs (flags, frontmatter, config maps passed in as args) and then does a single writeResource to persist an audit record of the decision. The decision needs no shared state; the write is append-only observability. But because that write rewrites the shared manifest under the lock, every such call contends with the workflow's lock hold.

Downstream effect: the caller has a local fallback for "swamp unavailable," so when the call times out (~60–90s) it silently falls back to a local computation and proceeds. Net result during any concurrent pipeline run:

  1. Every decision call pays a ~60–90s timeout before falling back, and
  2. The swamp-side decision/routing silently doesn't take effect — the local fallback wins by default. The behavior looks fine (no error surfaced to the user) but the swamp model's logic is effectively bypassed for the whole run.

So this isn't only a latency/fairness problem — for write-on-every-call audit patterns it can quietly nullify the model's intended behavior whenever another writer in the namespace holds the lock.

Why it strengthens the append-log direction

These audit/decision writes are the textbook case for option (1) in your post: they don't read-modify-write shared state, they only collide on the manifest. A commit-queue / append-log path (writer appends its delta and returns; a compactor coalesces) would let high-frequency decision/audit writes stop blocking — and stop being blocked into silent fallback — without weakening correctness for true RMW writers. A cheaper interim mitigation that would also help this class: let a write opt into "append-only, no manifest rewrite under lock" so audit artifacts don't serialize against unrelated work.

Happy to provide more concrete timing traces if useful. Filing as a ripple rather than a new issue per your note that the repro should live here.

stack72 commented 6/30/2026, 10:31:00 PM

@mgreten This should now be resolved. Two of the three directions you outlined have been addressed:

  1. Decouple the write from the manifest rewrite — shipped in #1720, which introduced a two-phase sync protocol (preparePush/commitPush). File uploads now happen outside the global lock, with only the index merge held under it. Both @swamp/s3-datastore and @swamp/gcs-datastore have shipped their implementations of the two-phase protocol.

  2. Incremental/sharded index — the partitioned _index/ shards were already in place; the two-phase sync was the missing piece to make the sync path honor that partitioning under the lock.

With (1) and (2) narrowing the critical section to just the index merge, lock holds should be short enough that the 60s timeout is no longer reachable under normal concurrency. If you still see contention after upgrading, post an update here mentioning @stack72 and we can reopen it — lock fairness/FIFO queuing (your direction 3) could be revisited as a follow-up.

mgreten commented 7/1/2026, 12:10:56 PM

@stack72 — thank you for shipping the two-phase sync so quickly, and for the clear writeup on #84. Upgraded the whole fleet and wanted to share what I'm seeing, in case the numbers are useful. Taking you up on the "post an update if you still see contention" offer — no urgency, and I'm glad to be wrong about any of this.

Setup: CLI 20260701.044223.0, @swamp/s3-datastore@2026.06.30.2, MinIO, on a single high-churn namespace whose .datastore-index.json is ~98 MB. I confirmed the two-phase path is actually active — the new markers show up under --verbose:

Preparing push (uploading files)...
Committing index update...
Committed 161 file(s) to datastore index

(One thing worth flagging for other users: the fix needs both the extension and a new enough core to dispatch — the extension alone with an older CLI silently stays on the single-phase path. Once I updated the CLI too, the markers appeared.)

What clearly improved: the file upload now happens outside the lock, exactly as designed. On a single write the Preparing push phase runs ~6–12s before Committing index update acquires the lock.

Where I still see contention: the Committing index update phase (commitPush, under the lock) takes ~26–35s on this index. A concurrent-writer test — 4 identical model-method writes to the same namespace at once — now lands at 2 succeed + 2 LockTimeout @60s, versus the pre-fix 1-winner + 3-timeouts. Genuinely better, but the 60s timeout is still reachable under normal concurrency here.

My read (very possibly incomplete): the critical section narrowed to the index merge, but on this namespace that merge is still proportional to the whole ~98 MB manifest — commitPush re-pulls, merges, and writes the full index back under the lock, so lock-hold is ~30s regardless of how small the actual write was. Only ~2 writers fit inside the 60s acquire window. The partitioned _index/ shards are present on the bucket (I can see the per-model JSON files), so if commitPush could scope the merge to just the touched shard rather than the monolith, that seems like it would close most of the remaining gap — which I think matches your own "the two-phase sync was the missing piece to make the sync path honor that partitioning" note.

Fully deferring to your judgment on whether that's the right layer, or whether the lock-fairness/FIFO direction (your #3) is the better next step to turn the hard 60s failure into latency. I don't want to presume a fix — just putting the measurements on the record. Happy to share the verbose phase-timing traces, the concurrent-writer harness, or the index/shard listing anytime.

Either way, upload-outside-the-lock is a real improvement and I appreciate it. Thanks again.

stack72 commented 7/1/2026, 5:55:18 PM

Shard-First Index Design Doc

A design doc for eliminating the monolithic .datastore-index.json has been written at design/shard-first-index.md in the core repo. It covers:

  • Motivation: mgreten's measurements showing 26-35s lock-hold in commitPush on a ~98 MB index
  • Target architecture: Extend partitioning to all datastore subdirectories, make _index/ shards the sole source of truth
  • Partition scheme: Per-model shards for data/, outputs/, definitions-evaluated/; per-workflow for workflow-runs/, workflows-evaluated/; single shard for low-cardinality dirs (audit/, telemetry/, etc.)
  • commitPush under the lock: Read/merge/write only the touched shard(s) — sub-second lock-hold
  • Zero-diff fast path: commitSeq counter in _meta.json replaces monolith ETag
  • Backward compatibility: Phase 1 dual-writes monolith + shards; phase 2 drops the monolith
  • Migration: First shard-first commit partitions the existing monolith; _meta.json version field gates the transition
  • Recovery: Self-healing _meta.json rebuild from S3 ListObjects

The implementation is entirely in the @swamp/s3-datastore extension — no core CLI changes needed. The design doc includes test dimensions and an implementation sequence.

PR with the design doc incoming.

stack72 commented 7/1/2026, 8:59:12 PM

Thanks @mgreten for reporting this! The fix has been merged and a release is on its way. We appreciate your contribution to swamp.

mgreten commented 7/2/2026, 12:07:45 PM

@stack72 — the shard-first work is exactly the fix this issue was hoping for (directions 1+2 combined), thank you. Upgraded the fleet to @swamp/s3-datastore 2026.07.01.1 + CLI 20260702.005505 and the design is great: explicit opt-in migrate-index, _meta.json v2 + commitSeq, and the monolith→shard partitioning itself ran in ~8s.

One heads-up on why I can't post the "4/4 concurrent writers" confirmation yet: on a namespaced repo the migration doesn't reach the live index. swamp datastore migrate-index migrates the bucket-root index rather than the repo's <namespace>/_index/, so our live agentic-tooling index stays v1/monolithic and commitPush keeps taking the ~60s monolith path. Filed the details separately as #913 so this issue stays focused on the shard-first design — just cross-linking here so the two are connected, since #913 is what's currently gating whether namespaced users actually see the #812 benefit.

Once #913 is sorted I'll migrate and re-run the concurrent-writer repro (was 1-winner + 3-timeouts here) and report the numbers back. Fully expecting the shard-first path to close it out — just wanted you to know the confirmation is pending the migration reaching the namespace, not pending the fix itself. Thanks again for turning this around so fast.

stack72 commented 7/2/2026, 3:27:02 PM

I am working on the namespace addition now @mgreten!

Sign in to post a ripple.