swamp serve is a single point of failure. Every piece of in-flight state — queued webhook runs, queued cron fires, active workflow executions, and worker dispatch bookkeeping — lives in memory. A process crash silently loses accepted work that callers already received a 200 queued for. On restart, a boot sweep tombstones interrupted workflows rather than recovering them. Running a second instance is actively destructive: the new instance's startup sweep destroys the first instance's live state.

This makes swamp serve unsuitable for any deployment where uptime matters — Kubernetes pods get replaced, ASG instances die, processes OOM, and hosts reboot. Today, every one of those events loses work with no recourse.

Desired Outcome

swamp serve can be deployed as one or more instances behind a load balancer with no accepted work lost on process death, no unsafe overlap during instance replacement, and no new infrastructure dependencies.

Specifically:

No accepted work is ever memory-only. Every webhook acknowledgement and cron fire is backed by a durable record before the caller is told it succeeded.
Process death is recoverable. Interrupted work is re-enqueued or resumed at startup according to a per-workflow policy, not silently discarded.
Instance replacement is safe. A new instance starting while an old one is still running cannot corrupt or destroy the old instance's in-flight state.
Multiple instances can share work. N instances can run against the same datastore, dividing work by per-item claims, with distributed deduplication for cron.
No new services. All coordination primitives (leases, epochs, claims, fire records) are records in the datastore the repo already has. The deployment unit remains a single swamp binary.

The full design brief and phase breakdown live in the artifact linked from this session. The implementation will be tracked as child issues of this one.

02Bog Flow

Closed

8/3/2026, 8:54:40 PM

No activity in this phase yet.

03Sludge Pulse

stack72 commented 7/28/2026, 11:23:32 PM

First PR - https://github.com/swamp-club/swamp/pull/2005

swamp serve ties a run's life to the WebSocket that started it. Socket dies, run dies. That makes HA impossible — there's nothing to reconnect to.

This PR adds --detach-runs. With it, runs execute independently of the client connection. A disconnected client can re-attach by run ID and pick up the event stream where it left off. Without the flag, behavior is identical to today.

this is the foundation for durable jobs, crash recovery and multi-instance so clients can reconnect

stack72 commented 7/29/2026, 10:20:15 PM

Second PR - https://github.com/swamp-club/swamp/pull/2012

You can now upgrade swamp serve without losing work.

With --detach-runs, stopping the process — whether it's a clean kill -TERM for an upgrade or an unexpected crash — no longer silently drops queued webhooks or leaves in-flight workflows stuck in limbo. Pending work is persisted to SQLite before the caller gets a 200. Interrupted runs are marked as failed with an honest reason tag so the operator can see exactly what happened and resume from the interrupted step.

Start the new binary, it reads the local state, replays pending work, and picks up where the old process left off. No manual intervention for the happy path.

stack72 commented 7/31/2026, 3:31:50 PM(edited)

PR 3 - https://github.com/swamp-club/swamp/pull/2036

swamp serve can now run on disposable infrastructure. When an ASG instance dies or a Kubernetes pod is replaced, the new instance reads heartbeats and pending work from S3, detects that its predecessor is dead, marks interrupted runs as failed, replays any queued webhooks, and starts serving — no operator involved.

stack72 commented 8/1/2026, 12:29:39 AM

PR 4 - https://github.com/swamp-club/swamp/pull/2038

We made multi-instance swamp serve stop double-firing cron workflows. When N instances share a schedule, they all race to create a fire record via putIfAbsent — one wins, the rest skip. The tricky part was deriving a deterministic key: croner's currentRun() is just new Date(), not the scheduled time, so we built canonicalFireTime() to snap wall-clock time back to the cron pattern grid. Best-effort by design — if the store is unreachable, it fires anyway rather than silently dropping work. A background reaper cleans up old records every 10 minutes.

stack72 commented 8/1/2026, 11:22:44 PM

PR 5 - https://github.com/swamp-club/swamp/pull/2042

Ensure that swamp serve can rehydrate from the datastore rather than a user needing to encode that as part of the init script. This helps keep the ephemeral nature of the instance

stack72 commented 8/2/2026, 9:40:02 PM

PR 6 - https://github.com/swamp-club/swamp/pull/2044

When one swamp serve instance dies in a multi-instance deployment, surviving instances now detect the dead peer within ~60 seconds and clean up its orphaned state — previously this only happened at boot, so stale heartbeats and runs could sit forever if no instance restarted. Detection timing, heartbeat interval, and stale threshold are all configurable via CLI flags.

stack72 commented 8/3/2026, 8:54:39 PM

All five desired outcomes from this issue are now shipped:

No accepted work is ever memory-only — pending runs persisted to SQLite + S3 control-plane store before ack (#2012, #2036)
Process death is recoverable — interrupted runs marked as failed with interrupt tags, pending work replayed on restart (#2012, #2036)
Instance replacement is safe — heartbeat-based stale detection, per-instance claims, boot + continuous reconciliation (#2036, #2044)
Multiple instances can share work — cron dedup via fire records (#2038), continuous reconciliation (#2044), cross-instance client re-attach (#1514), active run records for visibility (#1507)
No new services — all coordination is records in the control-plane store (S3). No Redis, Postgres, etcd, or broker.

PRs shipped:

#2005 — Phase 1: runs survive disconnection (--detach-runs)
#2012 — Phase 2: runs survive same-machine restart
#2017 — Control-plane store interface
#2033 — putIfAbsent CAS primitive
#2036 — Phase 3: disposable instance HA (heartbeat, identity, cross-machine reconciliation)
#2038 — Cron dedup via fire records
#2042 — Startup cache hydration
#2044 — Continuous reconciliation + per-instance claims
#1507 — Active run records in control-plane store
#1514 — Cross-instance client re-attach

Follow-on work tracked separately:

#1488 — Startup validation + /ready endpoint
#1505 — Auto-detect HA mode, deprecate --detach-runs
#1517 — Serve config file
Documentation: deployment guides, config reference, troubleshooting, architecture overview

Closing this issue — the HA system works end-to-end across all three deployment patterns (single instance, single instance with remote workers, N instances behind a load balancer).

Improve multi-word search query handling and guidance

swamp command doesn't check cwd

swamp auth login process fails when run remotely

serve boot OOM: findAllGlobalSince loads all workflow runs as full aggregates during reconciliation

extension search: channel-filtered results carry null version fields, and multiple --channel values collapse instead of unioning

s3-datastore: localHasAllRemoteEntries uses bare rel instead of localRelPath, always returns false in namespaced mode

datastoreSetupExtension migrates .swamp/ to cache root instead of namespace directory

swamp serve writes outputs/ outside the bound namespace — silently never persisted to the remote datastore

s3-datastore: bulk pull writes to cache root instead of the namespace, wedging push and arming a full-datastore delete

git: add pull, fetch --tags, and cherry-pick methods

Workflow runs with wide dataArtifact fan-out (~400/run) OOM both workflow history get and swamp serve boot, crash-looping the scheduler; rollback doesn't help (state-triggered, not release-triggered)

Show user:<name> instead of user:<id> in grant list output

Reusable long-lived invite link: one URL to bring people into the swamp

Exclude swamp internal model types from model list output

Skill docs use invalid two-argument 'vault get' form

access token mint prints an invalid retrieval command (vault get, not vault read-secret)

serve reload and model edit timeout over --server in Kubernetes

swamp auth whoami should report paid tier and trial status

Use whoami entitlement to explain private-extension push refusals

Document --restricted-commands flag for swamp serve

vault put: pass tags through to extension provider CreateSecret/setSecret calls

Docs: update manual for 23 new --server commands

Update VaultProvider interface in @systeminit/swamp-testing to support optional tags in put()

Docs: document --grants-file flag for swamp serve

Docs: document --server support for serve reload

Docs: add --tag flag to vault put reference

Docs: update manual for --detach-runs deprecation and HA auto-detection

Document the Codex plugin in swamp docs

Add --server support to CLI commands missing it

Document the swamp Claude Code plugin in the swamp documentation

Make --server a global CLI option

Add --server support to model cancel (server-side run cancellation)

Load grants from external source on swamp serve startup

swamp vault put should support tags for secret governance

Improve hot reload of the deamon, include extension source/bundle changes

Authenticated users must re-login after swamp serve restart

Add --server support to serve reload

Harden queue claim semantics: atomic claims + status-guarded markDelivered/markFailed across app queues

Auto-detect HA mode and deprecate --detach-runs flag

version-drift check (from #236) is a raw string-inequality, false-positives on multi-model manifests and metadata-only bumps

Resend newsletter audience sync: contacts, segments, backfill, unsubscribe sync-back

Move per-login token secrets from vault to encrypted control-plane store

Docs: document macOS TLS trust limitation and SSL_CERT_FILE workaround

Validate swamp serve deployment stack on startup

Docs: serve config file (.swamp/serve.yaml) and --config flag

Serve config file — read swamp serve configuration from YAML

Deno runtime does not trust ISRG Root YR — extensions fail with valid Let's Encrypt certs

Docs: update hot-reload reference to include trust list refresh

Cross-instance client re-attach for multi-instance swamp serve

vercel codegen: inline rawSyncResult in non-unwrap sync path

vercel codegen: unused listAll import blocks extension publish for 4 models

vercel codegen: add round-trip idempotency test for the pipeline

Operatives button on Collectives page does nothing

Track active runs in the control-plane store for cross-instance visibility

validateGlobalArgInputRefs checks step-level inputs but step schema rejects 'inputs' key

Docs: document continuous reconciliation and multi-instance serve flags

Docs: update workflow guard skip output to include guardExpression and guardResult

Make it aware hot reload on the deamon does not take into account trust list changes, or do take into account the trust list for the hot reload.

workflow validate: new 'GlobalArgument input references' check resolves model-definition ${{ inputs.* }} against workflow inputs, false-failing steps that supply the input

Continuous reconciliation for multi-instance swamp serve

Split the score-read circuit breaker so leaderboard pressure can't blank profile scores

oauth_sign_in telemetry records provider as the literal ":id" for every event

Ghost profile bypasses the score-read seam and 500s on a slow ledger

Profile score section blocks the render on a ClickHouse read

Council: set the scoring balance — command tiers, farm-decay parameters, cross-category ratio

Scoring coverage gaps: instrument drop-off, score ripples/mentions, reward invite-converts

Push down the two cheap leaderboard scans: ghost-gate dictionary and eventCounts owner filter

Materialize an active-days-by-id table for the streak board

score-rollup-projector: O(all-time ledger) closure + per-pod fan-out saturates prod ClickHouse CPU

Startup cache hydration for swamp serve instance replacement

Output a log line for steps skipped due to guards

extension quality scorer misreads a template-literal import specifier in generated code as an unresolvable bare import, hard-erroring the whole extension as UNSCORABLE

Add @swamp/kubernetes/statefulset model type

Docs: access token commands reference says plaintext is 'shown once'

Document --restricted-model-types flag for swamp serve

swamp auth server-login ignores SWAMP_SERVE_URL

s3-datastore extension: pullChanged writes namespace-stripped paths to local cache

serve oauth: an unrecognized device-token error kills the whole serve process instead of the bootstrap

s3-datastore: bidirectional sync pull writes namespace-stripped copies to cache root, wedging push with no supported recovery

CLI reports 'HTTP/2 not supported by this client' when the server actually returned 401 — masks auth failures as a transport error