swamp serve closes a connection's WebSocket when Deno's WS keepalive gets no pong in time, and there is no way to tune the ping interval or pong timeout — no serve flag, no workflow run flag, no SWAMP_* env (workflow run --timeout is only a run cancellation deadline). serve calls Deno.upgradeWebSocket(req) with NO options (src/cli/commands/serve.ts:898/902), so the liveness behavior is Deno's hardcoded default.

The failure mode: whenever serve's single JS event loop is blocked longer than that timeout, it can't process incoming pong frames on ANY connection, so Deno's keepalive fires on all of them at the same instant and serve closes them:

serve·connection  WebSocket error: "No response from ping frame."   (×N — every live connection, same millisecond)

Because the run is coupled to the connection, the in-flight placed methods then fail (Cannot push to a closed AsyncQueue), the run aborts mid-flight, and downstream teardown is skipped so provisioned resources leak (in our case 7 worker containers, reaped by hand).

TWO CORRECTIONS to earlier versions of this report:

An early draft blamed client-side CPU starvation / worker cold-start spikes. Wrong — the peers (workers + client) were healthy and ponging; serve simply couldn't read them while its own loop was blocked. The host was a 12-core box, orchestrator uncapped, load ~5.5 — not CPU-bound.

A later draft blamed deno bundle: it claimed serve rebuilt the fan-out type's ship-bundle once per instance (~8.4s each) because bundleSourceFactory is memoized per model-definition. That is ALSO wrong, on both counts. Verified directly: (a) deno bundle of the real extension (a 113-line file that transitively imports a 3,565-line model + its full dep graph) is 88ms warm, not 8.4s; (b) bundleSourceFactory is memoized per type (bundlePromise ??= in model_kind_adapter.ts at the registerExtensionType / promoteFromLazy site), so N instances of one type bundle exactly once. Bundling is not what blocks the loop.

What actually blocked the loop (~60–70s in our case): a forEach fanned out to 7 model instances, all dispatched at the same instant — but serve resolved each step's model definition serially, ~10s apart, back-to-back, with no yield in between. The per-step model·method·run … Found model … events land like clockwork at +10.2s, +9.8s, +9.9s, +9.95s, +9.97s, +9.86s. 7 × ~10s ≈ 70s of contiguous loop occupancy > the keepalive window, so at the 60s mark every connection (7 workers + the client) dropped in the same millisecond.

The ~10s is consumed BEFORE each "Found model" line — i.e. in the definition resolution itself: execution_service.ts → findDefinitionByIdOrName → definitionRepo.findByNameGlobal(name) (model_lookup.ts:131). That global definition lookup appears to be O(store) per call and runs on serve's single event loop, so a fan-out of N steps serializes into N × lookup-cost of uninterrupted loop time. (We have not micro-profiled findByNameGlobal itself — the ~10s/step is measured end-to-end from the run log — but the gap is squarely in the resolve-definition step, not in bundling or in worker execution.)

The specific slow path is a separate concern worth its own issue; the point for THIS issue is that any multi-second serve-loop occupancy — a serial fan-out resolve, a large dispatch prep, GC — silently kills every live connection because the keepalive is un-tunable AND nothing yields the loop to service pings during the block.

Proposed Solution

Make the WebSocket liveness timeout explicit and configurable (defaults preserved):

The minimal fix: pass an explicit idleTimeout to the two Deno.upgradeWebSocket(req) calls (serve.ts:898/902) instead of inheriting Deno's default, and make it configurable:
- swamp serve --ws-idle-timeout <duration> (env SWAMP_WS_IDLE_TIMEOUT).
Optionally split into --ws-ping-interval / --ws-pong-timeout if finer control is wanted.

Raising the timeout lets a transiently-blocked serve loop recover without dropping healthy connections, and lets operators confirm the behavior (set it generous, re-run, watch the long fan-out survive).

NOTE: a generous timeout is a BACKSTOP, not the whole fix. The deeper fixes are orthogonal and both worth doing: (1) stop occupying the loop for N × lookup — resolve fan-out step definitions concurrently and/or make findByNameGlobal not O(store) per call; (2) decouple the run from the connection so a transient drop can't abort an in-flight run (Lab #519). But a configurable keepalive is valuable independently: serve should not abort a run because its own loop was briefly busy.

Affected Components

serve WebSocket setup — pass an explicit, configurable idleTimeout to Deno.upgradeWebSocket (serve.ts:898/902) instead of relying on the runtime default.
swamp serve flag + env plumbing.
Docs — list the new flag/env alongside the existing serve options.

Why It Matters

serve's event loop will occasionally be busy for several seconds (serial fan-out definition resolution, large dispatch prep, GC). With an un-tunable, runtime-default keepalive, any such pause silently drops every connected worker AND the client at once, fails the in-flight methods, and leaks whatever the run provisioned. Long-running self-hosted fan-out workflows — exactly what serve + remote execution exist for — are the most exposed. A configurable timeout makes the system resilient to its own brief stalls.

The tactical complement to detached / resumable runs (Lab #519): #519 removes the run↔connection coupling so a drop can't abort a run; this makes the drop far less likely in the first place.
The serial fan-out definition-resolve cost (~10s/step via findByNameGlobal on the serve loop) is the trigger we hit and deserves its own performance issue; this issue is specifically about the keepalive being un-tunable so that ANY loop stall is fatal.

02Bog Flow

Shipped

6/27/2026, 12:28:39 AM

Click a lifecycle step above to view its details.

03Sludge Pulse

stack72 assigned stack726/26/2026, 11:29:55 PM

stack72 commented 6/27/2026, 12:29:24 AM

Thanks @swamp_lord for reporting this! The fix has been merged and a release is on its way. We appreciate your contribution to swamp.

Improve multi-word search query handling and guidance

swamp command doesn't check cwd

swamp auth login process fails when run remotely

serve boot OOM: findAllGlobalSince loads all workflow runs as full aggregates during reconciliation

extension search: channel-filtered results carry null version fields, and multiple --channel values collapse instead of unioning

s3-datastore: localHasAllRemoteEntries uses bare rel instead of localRelPath, always returns false in namespaced mode

datastoreSetupExtension migrates .swamp/ to cache root instead of namespace directory

swamp serve writes outputs/ outside the bound namespace — silently never persisted to the remote datastore

s3-datastore: bulk pull writes to cache root instead of the namespace, wedging push and arming a full-datastore delete

git: add pull, fetch --tags, and cherry-pick methods

Workflow runs with wide dataArtifact fan-out (~400/run) OOM both workflow history get and swamp serve boot, crash-looping the scheduler; rollback doesn't help (state-triggered, not release-triggered)

Show user:<name> instead of user:<id> in grant list output

Reusable long-lived invite link: one URL to bring people into the swamp

Exclude swamp internal model types from model list output

Skill docs use invalid two-argument 'vault get' form

access token mint prints an invalid retrieval command (vault get, not vault read-secret)

serve reload and model edit timeout over --server in Kubernetes

swamp auth whoami should report paid tier and trial status

Use whoami entitlement to explain private-extension push refusals

Document --restricted-commands flag for swamp serve

vault put: pass tags through to extension provider CreateSecret/setSecret calls

Docs: update manual for 23 new --server commands

Update VaultProvider interface in @systeminit/swamp-testing to support optional tags in put()

Docs: document --grants-file flag for swamp serve

Docs: document --server support for serve reload

Docs: add --tag flag to vault put reference

Docs: update manual for --detach-runs deprecation and HA auto-detection

Document the Codex plugin in swamp docs

Add --server support to CLI commands missing it

Document the swamp Claude Code plugin in the swamp documentation

Make --server a global CLI option

Add --server support to model cancel (server-side run cancellation)

Load grants from external source on swamp serve startup

swamp vault put should support tags for secret governance

Improve hot reload of the deamon, include extension source/bundle changes

Authenticated users must re-login after swamp serve restart

Add --server support to serve reload

Harden queue claim semantics: atomic claims + status-guarded markDelivered/markFailed across app queues

Auto-detect HA mode and deprecate --detach-runs flag

version-drift check (from #236) is a raw string-inequality, false-positives on multi-model manifests and metadata-only bumps

Resend newsletter audience sync: contacts, segments, backfill, unsubscribe sync-back

Move per-login token secrets from vault to encrypted control-plane store

Docs: document macOS TLS trust limitation and SSL_CERT_FILE workaround

Validate swamp serve deployment stack on startup

Docs: serve config file (.swamp/serve.yaml) and --config flag

Serve config file — read swamp serve configuration from YAML

Deno runtime does not trust ISRG Root YR — extensions fail with valid Let's Encrypt certs

Docs: update hot-reload reference to include trust list refresh

Cross-instance client re-attach for multi-instance swamp serve

vercel codegen: inline rawSyncResult in non-unwrap sync path

vercel codegen: unused listAll import blocks extension publish for 4 models

vercel codegen: add round-trip idempotency test for the pipeline

Operatives button on Collectives page does nothing

Track active runs in the control-plane store for cross-instance visibility

validateGlobalArgInputRefs checks step-level inputs but step schema rejects 'inputs' key

Docs: document continuous reconciliation and multi-instance serve flags

Docs: update workflow guard skip output to include guardExpression and guardResult

Make it aware hot reload on the deamon does not take into account trust list changes, or do take into account the trust list for the hot reload.

workflow validate: new 'GlobalArgument input references' check resolves model-definition ${{ inputs.* }} against workflow inputs, false-failing steps that supply the input

Continuous reconciliation for multi-instance swamp serve

Split the score-read circuit breaker so leaderboard pressure can't blank profile scores

oauth_sign_in telemetry records provider as the literal ":id" for every event

Ghost profile bypasses the score-read seam and 500s on a slow ledger

Profile score section blocks the render on a ClickHouse read

Council: set the scoring balance — command tiers, farm-decay parameters, cross-category ratio

Scoring coverage gaps: instrument drop-off, score ripples/mentions, reward invite-converts

Push down the two cheap leaderboard scans: ghost-gate dictionary and eventCounts owner filter

Materialize an active-days-by-id table for the streak board

score-rollup-projector: O(all-time ledger) closure + per-pod fan-out saturates prod ClickHouse CPU

Startup cache hydration for swamp serve instance replacement

Output a log line for steps skipped due to guards

extension quality scorer misreads a template-literal import specifier in generated code as an unresolvable bare import, hard-erroring the whole extension as UNSCORABLE

Add @swamp/kubernetes/statefulset model type

Docs: access token commands reference says plaintext is 'shown once'

Document --restricted-model-types flag for swamp serve

swamp auth server-login ignores SWAMP_SERVE_URL

s3-datastore extension: pullChanged writes namespace-stripped paths to local cache

serve oauth: an unrecognized device-token error kills the whole serve process instead of the bootstrap

s3-datastore: bidirectional sync pull writes namespace-stripped copies to cache root, wedging push with no supported recovery

CLI reports 'HTTP/2 not supported by this client' when the server actually returned 401 — masks auth failures as a transport error