On a single high-churn namespace, datastore write throughput is gated by something downstream of the per-namespace lock: every write pulls and pushes the whole index under the lock, so lock-hold scales with total index size rather than with the partition a write touches. For a workload that fans many concurrent writers into one namespace, this serializes everything and produces 60s lock-acquire timeouts.

Per-namespace locking (#666, shipped) solved contention between namespaces and is great. This is the remaining intra-namespace concurrency problem, split out from #666 since that issue is marked shipped. Filing it on its own so it can be tracked, with the full workload context that I don't think was ever on the record.

The workload (the part that wasn't communicated before)

I run an automated dev-workflow system. The shape that matters for the datastore:

Many pipeline runs execute concurrently, each in its own git worktree (isolated filesystem, branch, ports).
Each run is event-driven: it historically emitted a datastore write on every phase transition and (for some subscribers) every agent call — so a single run produced dozens of writes.
All of those writes funnel into one namespace. The worktrees isolate the filesystem, but every run points its datastore calls at the same control-plane repo → same namespace → same lock. Worktree isolation is filesystem isolation, not datastore isolation.
Separately, ~13 scheduled jobs on one machine plus a second machine write that same namespace continuously.

So per-namespace locking doesn't help this workload: the entire concurrent fan-out lives inside one namespace and serializes on its lock.

Measurements

Sampling the lock object every 2s and attributing each hold to its command, on aligned latest versions:

Lock-hold p50 ≈ 94–102s, max 268s, across a ~20-minute window.
The longest holds were spread across completely different operations — a notification send, a small provider lookup, a poller refresh — all converging on the same ~268s ceiling. That uniformity is the tell: the duration tracks the shared whole-index sync, not what the operation does.
Index was ~93 MB; a single high-cardinality data stream was ~40% of it, inflating the hold for every writer.

Concrete impact: a batch of back-to-back runs that should complete in roughly the low single-digit hours instead ran overnight — the wall-clock was dominated by writers waiting on the lock, not by the actual work.

What I changed on my side (so this is scoped to what's genuinely the datastore's)

I've already removed the avoidable share of this:

Collapsed a per-invocation unique-named artifact (resolve-<phase>-<timestamp>) to a stable per-phase name — it was thousands of write-only index entries nothing read back.
Aligned the write unit with the run unit: telemetry-class writes are now buffered in order during a run and replayed as a single batch at completion, so one run holds the lock once instead of dozens of times.

Those cut the number of lock acquisitions dramatically. But the per-acquisition cost (whole-index sync) is unchanged, so concurrent runs and the scheduled writers still serialize on the single lock — that part is the datastore's to solve, which is why I'm filing it.

Two directions that would each independently unblock fan-out workloads

Thinking out loud, not prescribing:

Incremental / scoped index sync — if a write synced only the partition it touched rather than the whole index, same-namespace concurrent writers would stop blocking each other for tens of seconds. This is the more general fix and helps every at-scale user, not just fan-out ones. (The partitioned _index/ shards already exist; this would be making the sync under the lock honor that partitioning.)
A lightweight per-run / ephemeral namespace primitive — let a short-lived job cheaply get its own lock scope and fold its data into a parent namespace afterward, without standing up a separate repo checkout and recreating model instances by hand. Today the only way to get a second lock is a second checkout, which is too heavy for a per-run pattern.

What I'm considering if neither lands

If intra-namespace concurrency stays serialized, I'll likely have to break my workload apart at the repo level — a separate checkout (hence separate namespace + lock) per writer-class (pollers vs. pipeline vs. the second machine), and possibly an ephemeral per-run namespace that I provision and tear down myself, harvesting each run's data into a central analytics namespace afterward. That works (my analytics already reassembles from an on-disk source of truth, so runs can live in any namespace), but it's a lot of self-managed namespace plumbing to work around the lock — exactly the kind of thing a primitive like (2), or simply (1), would make unnecessary.

Happy to share the lock-sampling script, the per-run write trace, or profiling data if any of it would help. Thanks again for all the recent datastore work — #788 and the per-namespace locks have both been real improvements even as I work through this.

Environment

@swamp/s3-datastore@2026.06.24.1
swamp 20260625.225837.0
MinIO backend, two writer machines sharing one bucket

02Bog Flow

Shipped

6/30/2026, 12:31:30 PM

Click a lifecycle step above to view its details.

03Sludge Pulse

stack72 assigned stack726/30/2026, 10:13:47 AM

mgreten commented 6/27/2026, 3:29:55 PM

Following up with a concrete workload that makes this issue's impact tangible, in case it helps prioritize.

I'm exploring "sandbox" isolation for the fan-out described above: spinning up N parallel agentic-tooling work-streams, each ideally with its own datastore lock so they stop serializing on the one namespace lock. While prototyping I confirmed the current levers and where they fall short — sharing in case the framing is useful:

What works today (and is genuinely great):

Per-namespace locking (#666) cleanly isolates between namespaces — verified .locks/<ns>.lock per namespace.
datastore catalog pull --namespaces <ns> + cross-namespace CEL (#516) give a solid read/merge-back path: I pulled a foreign namespace's full catalog (5,283 rows) into another repo's view in one call. This means a "harvest cold" merge model is buildable in userland.
A runtime prefix via SWAMP_DATASTORE gives a fully isolated datastore root + its own lock with zero .swamp.yaml edits — nice for throwaway sandboxes.

Where the workaround gets fiddly (the gap that keeps pointing back at #829):

The cheap isolation lever (env-var prefix) and the blessed merge-back lever (catalog pull, which reads <namespace>/.catalog-export.json at the top level) don't line up — a prefix-isolated sandbox nests as <prefix>/<namespace>/…, so catalog pull can't reach it. To get native retrieval I have to make each sandbox a real namespace, which means a .swamp.yaml per sandbox.
And even with namespace-per-sandbox, this is all working around the actual problem: the hot fan-out still can't share one namespace because of the whole-index-sync-under-lock behavior this issue describes. The sandbox scheme is essentially me partitioning the workload by hand to dodge that.

No pressure on timeline at all — I know there's a lot in flight. Mostly wanted to put the real use-case on the record: if intra-namespace writes could avoid holding the lock for the whole index (the append/partition idea in #812), the entire sandbox workaround becomes unnecessary. Happy to share lock-hold measurements or test a build against this workload whenever it's useful.

stack72 commented 6/30/2026, 12:31:39 PM

Thanks @mgreten for reporting this! The fix has been merged and a release is on its way. We appreciate your contribution to swamp.

mgreten commented 7/6/2026, 11:02:15 PM

Following up with fresh measurements on @swamp/s3-datastore 2026.07.02.1 (the shard-first release), because I'm still seeing the whole-index-sync cost this issue describes — and this time it's not contention, which surprised me.

The measurement (fully quiesced, single writer)

I stopped every scheduled writer on the namespace, killed all in-flight swamp-bin processes, confirmed the lock was free, then timed two back-to-back writes of a method whose actual work is sub-second (a pr-test-planner planForPr that early-outs on a backend-only PR — no LLM call, tiny artifact):

run 1: WALL=1159.1s  method=719ms  status=succeeded
run 2: WALL=1251.6s  method=767ms  status=succeeded

So ~19–21 minutes wall per write, of which <1s is the method. The remaining ~1150–1250s is pure datastore sync, paid by a single uncontended writer. Run 2 wasn't faster than run 1, so it isn't cold-cache/first-touch — it's a fixed per-write cost that tracks total namespace size, not the shard the write touches.

Namespace at the time: 123,797 catalog rows / ~95 MB, dominated by two streams — adw-analytics sessions (62,222 rows, lifetime: infinite) and an orphaned pr-feed stream (44,389 rows left behind by a model that migrated to another namespace). The write above touches only pr-test-planner (166 rows). It should not be paying for the other 123k.

Why this looks like the shard-first fast path not engaging

The 2026.07.02.1 description advertises "shard-first indexing with per-model/per-workflow partition shards under _index/ as the source of truth" and a "commitSeq-based zero-diff fast path." But:

Two identical consecutive writes both pay the full ~20 min — a commitSeq zero-diff path should make the second nearly free.
A write touching one 166-row model still pays for a 62k-row sibling stream — per-shard sync should skip untouched shards.

One thing I noticed while poking (offered as a lead, not a conclusion): the local cache under .swamp/data/ is a single monolithic _catalog.db — there's no local _index/ shard directory. If the partition shards are the source of truth in S3 but the local side still reconciles a monolithic catalog on every write, that would explain why the per-shard layout doesn't reduce the per-write cost. I could be misreading the layout, though.

What I've already done on my side

Following the guidance in this issue, I've kept my own write amplification down (batched telemetry, stable artifact names, cut a per-PR snapshot retention from 100×30d to 5×7d). Those help the write count. But the per-acquisition cost above is what still dominates, and it's what I think is squarely the datastore's.

Happy to share the lock-sampling script, the two-write repro harness, or run a build against this namespace whenever it's useful — this one has a nicely pathological 95 MB index to test against. And thank you again for the shard-first work; #666 and the two-phase sync have both been real improvements even as I chase this last piece.

Environment

@swamp/s3-datastore 2026.07.02.1
swamp 20260706.015952.0-sha.a3f4a37c
MinIO backend, single namespace, single writer during the measurement

mgreten commented 7/7/2026, 2:18:00 AM

Splitting my two follow-ups above into standalone issues so they can be triaged on their own — they're really separate from this issue's concurrency/fan-out scope:

#1008 — datastore migrate-index aborts on a hardcoded 30s S3 requestTimeout for large shards (the intended monolithic→shard-first fix can't complete at scale).
#1009 — orphaned data (from a migrated-away/deleted model) can't be GC'd or deleted.

What remains squarely on this issue: my quiesced-write measurement (two back-to-back single-writer writes, ~1159s and ~1252s wall for <1s of method work, on a 95MB/630k-entry namespace whose local index is still monolithic). That's the whole-index-sync cost this issue names, still present on 2026.07.02.1 — offered as evidence in case it's useful for deciding whether this stays closed. No pressure either way; thanks again.

mgreten commented 7/7/2026, 9:41:34 PM

One more facet from the same root cause, on the interactive-UX side — flagging because it's what actually blocked a live agent, and it's arguably separable from the throughput fix.

When the whole-index sync is slow (the ~20min-per-write cost this issue describes, on a ~95MB namespace), an interactive swamp model method run … gives the caller no backpressure: it sits on "Syncing model from datastore…" for many minutes with no progress output, no ETA, and no fail-fast. A live agent (in another repo, calling a swamp model to file a ticket) hung 7+ minutes on that line and was ultimately SIGKILLed by its own 2-minute tool timeout — so from the caller's side it's indistinguishable from a hang, and it leaves them retrying into the same wall.

Even before the throughput fix lands, two things would make this far more livable for interactive callers:

Progress / heartbeat on the sync line (e.g. "syncing index: N/M shards" or bytes transferred) so a caller can tell it's working vs. wedged.
A fail-fast option — a client-visible sync deadline that returns a clear "datastore busy, try later / lock contended" error instead of an open-ended hang. Today SWAMP_DATASTORE_SYNC_TIMEOUT_MS exists but defaults high and, when it does fire, surfaces as a lock_timeout rather than actionable backpressure.

Not asking to reprioritize the core fix (#1008 migrate-index + shrinking the index is the real cure) — just noting that the hang-with-no-signal behavior is a distinct pain that a cheaper progress/deadline change could relieve in the meantime. Happy to share the caller-side transcript if useful.

mgreten commented 7/8/2026, 5:17:32 PM

Wanted to close the loop with good news and a thank-you: we migrated to shard-first v2 today (@swamp/s3-datastore 2026.07.02.1) and the scoped pull is excellent — pullIndex ~423ms on a namespace where reads used to take minutes. The remaining cost we see is on the push side (the v2 path still re-uploads the full monolithic back-compat index + catalog export each commit); we've written that up with traces and measurements in #1034 rather than piling onto this one. Thanks for landing this — it made a real difference for us.

Improve multi-word search query handling and guidance

swamp command doesn't check cwd

swamp auth login process fails when run remotely

serve boot OOM: findAllGlobalSince loads all workflow runs as full aggregates during reconciliation

extension search: channel-filtered results carry null version fields, and multiple --channel values collapse instead of unioning

s3-datastore: localHasAllRemoteEntries uses bare rel instead of localRelPath, always returns false in namespaced mode

datastoreSetupExtension migrates .swamp/ to cache root instead of namespace directory

swamp serve writes outputs/ outside the bound namespace — silently never persisted to the remote datastore

s3-datastore: bulk pull writes to cache root instead of the namespace, wedging push and arming a full-datastore delete

git: add pull, fetch --tags, and cherry-pick methods

Workflow runs with wide dataArtifact fan-out (~400/run) OOM both workflow history get and swamp serve boot, crash-looping the scheduler; rollback doesn't help (state-triggered, not release-triggered)

Show user:<name> instead of user:<id> in grant list output

Reusable long-lived invite link: one URL to bring people into the swamp

Exclude swamp internal model types from model list output

Skill docs use invalid two-argument 'vault get' form

access token mint prints an invalid retrieval command (vault get, not vault read-secret)

serve reload and model edit timeout over --server in Kubernetes

swamp auth whoami should report paid tier and trial status

Use whoami entitlement to explain private-extension push refusals

Document --restricted-commands flag for swamp serve

vault put: pass tags through to extension provider CreateSecret/setSecret calls

Docs: update manual for 23 new --server commands

Update VaultProvider interface in @systeminit/swamp-testing to support optional tags in put()

Docs: document --grants-file flag for swamp serve

Docs: document --server support for serve reload

Docs: add --tag flag to vault put reference

Docs: update manual for --detach-runs deprecation and HA auto-detection

Document the Codex plugin in swamp docs

Add --server support to CLI commands missing it

Document the swamp Claude Code plugin in the swamp documentation

Make --server a global CLI option

Add --server support to model cancel (server-side run cancellation)

Load grants from external source on swamp serve startup

swamp vault put should support tags for secret governance

Improve hot reload of the deamon, include extension source/bundle changes

Authenticated users must re-login after swamp serve restart

Add --server support to serve reload

Harden queue claim semantics: atomic claims + status-guarded markDelivered/markFailed across app queues

Auto-detect HA mode and deprecate --detach-runs flag

version-drift check (from #236) is a raw string-inequality, false-positives on multi-model manifests and metadata-only bumps

Resend newsletter audience sync: contacts, segments, backfill, unsubscribe sync-back

Move per-login token secrets from vault to encrypted control-plane store

Docs: document macOS TLS trust limitation and SSL_CERT_FILE workaround

Validate swamp serve deployment stack on startup

Docs: serve config file (.swamp/serve.yaml) and --config flag

Serve config file — read swamp serve configuration from YAML

Deno runtime does not trust ISRG Root YR — extensions fail with valid Let's Encrypt certs

Docs: update hot-reload reference to include trust list refresh

Cross-instance client re-attach for multi-instance swamp serve

vercel codegen: inline rawSyncResult in non-unwrap sync path

vercel codegen: unused listAll import blocks extension publish for 4 models

vercel codegen: add round-trip idempotency test for the pipeline

Operatives button on Collectives page does nothing

Track active runs in the control-plane store for cross-instance visibility

validateGlobalArgInputRefs checks step-level inputs but step schema rejects 'inputs' key

Docs: document continuous reconciliation and multi-instance serve flags

Docs: update workflow guard skip output to include guardExpression and guardResult

Make it aware hot reload on the deamon does not take into account trust list changes, or do take into account the trust list for the hot reload.

workflow validate: new 'GlobalArgument input references' check resolves model-definition ${{ inputs.* }} against workflow inputs, false-failing steps that supply the input

Continuous reconciliation for multi-instance swamp serve

Split the score-read circuit breaker so leaderboard pressure can't blank profile scores

oauth_sign_in telemetry records provider as the literal ":id" for every event

Ghost profile bypasses the score-read seam and 500s on a slow ledger

Profile score section blocks the render on a ClickHouse read

Council: set the scoring balance — command tiers, farm-decay parameters, cross-category ratio

Scoring coverage gaps: instrument drop-off, score ripples/mentions, reward invite-converts

Push down the two cheap leaderboard scans: ghost-gate dictionary and eventCounts owner filter

Materialize an active-days-by-id table for the streak board

score-rollup-projector: O(all-time ledger) closure + per-pod fan-out saturates prod ClickHouse CPU

Startup cache hydration for swamp serve instance replacement

Output a log line for steps skipped due to guards

extension quality scorer misreads a template-literal import specifier in generated code as an unresolvable bare import, hard-erroring the whole extension as UNSCORABLE

Add @swamp/kubernetes/statefulset model type

Docs: access token commands reference says plaintext is 'shown once'

Document --restricted-model-types flag for swamp serve

swamp auth server-login ignores SWAMP_SERVE_URL

s3-datastore extension: pullChanged writes namespace-stripped paths to local cache

serve oauth: an unrecognized device-token error kills the whole serve process instead of the bootstrap

s3-datastore: bidirectional sync pull writes namespace-stripped copies to cache root, wedging push with no supported recovery

CLI reports 'HTTP/2 not supported by this client' when the server actually returned 401 — masks auth failures as a transport error