Two related memory-exhaustion failures on a repo whose workflow runs carry wide dataArtifact fan-out (~400 artifacts per run), both V8 "Fatal JavaScript out of memory: Ineffective mark-compacts near heap limit" at a ~4 GB heap:

swamp workflow history get <workflow> --json OOMs. 415 accumulated runs whose on-disk records total only 22 MB blow a 4 GB heap (~185x amplification), so the assembly path is materializing far more than the input. Sibling of #1173 (history search, shipped) — reproduced on 20260807.031228.0, i.e. after that fix, so history get was not covered.
swamp serve OOM-crash-loops at BOOT on the same repo — dying immediately after Policy snapshot loaded (the next, unlogged boot phase), restart-looping forever (6+ consecutive identical crashes). Critically:
- reproduces on 20260807.031228.0 and on 20260728.192016.0 — the exact binary that had served this repo continuously for the previous 2.5 weeks (the first failing boot was simply the first restart in weeks);
- reproduces with the workflow-runs directory trimmed from 416 run files to 20 — so it is NOT the .swamp/workflow-runs scan;
- a plain CLI in the same container (swamp model method run, swamp workflow run, full repo init included) works throughout — the failure is specific to a serve boot phase.
Net effect: a scheduled workflow with wide per-run artifact fan-out eventually bricks its own scheduler — one day serve simply cannot boot its repo any more, and rolling back the binary does not help because the trigger is accumulated state, not the release.

Environment

swamp 20260807.031228.0-sha.9a36314e and 20260728.192016.0-sha.f2a1f6b9 (Linux container, scheduler-only serve: --host 127.0.0.1 --auth-mode none, init: true)
Datastore @keeb/mongodb-datastore (namespaced); vault @webframp/hashicorp-vault
Workflow: 4 model-method steps per run, hourly schedule since 2026-07-20; one step fans out over ~400 targets, so each run records ~400+ dataArtifacts
.swamp/workflow-runs/<workflow-id>/: 415 files, 22 MB; repo data tree ~302k files / ~4 GB

Steps to reproduce

Schedule a workflow whose steps produce hundreds of dataArtifacts per run; let it run hourly for a few weeks.
swamp workflow history get <name> --json → OOM.
Restart the serve container → boot OOM crash-loop.

Observed

history get on 20260728 (~93 s in):

[4682:0x555ae8c9a000]    91933 ms: Scavenge (during sweeping) 4083.0 (4086.1) -> 4082.2 (4086.4) MB ... allocation failure;
# Fatal JavaScript out of memory: Ineffective mark-compacts near heap limit

(on 20260807 the same command dies as a bare exit-137 kill with zero output.)

serve boot (both versions, every restart, last logged line before death):

[INF] serve: HA: detached runs, pending-run durability (no remote control-plane — heartbeat and reconciliation disabled)
[INF] serve: Control-plane store: local filesystem fallback
[INF] swamp·domain·access·policy-snapshot: Loaded policy snapshot: 0 active grant(s), 0 group(s)
[INF] serve: Policy snapshot loaded (reload mode: "manual")
# Fatal JavaScript out of memory: Ineffective mark-compacts near heap limit

Expected / suggestions

history get should load only the most recent run by default and paginate beyond that (mirror of whatever #1173 did for search); a --last N bound for the whole history family.
Whichever serve boot phase follows the policy snapshot needs the same bounding — it currently scales with accumulated run/artifact state until boot is impossible. A log line naming that phase would also make this diagnosable in the field (the crash is currently silent about what it was doing).
The ~185x blow-up (22 MB of run records -> 4 GB heap) suggests per-run dataArtifact expansion is held in memory all at once; worth profiling independently.

Workaround in production

Serve disabled (command: sleep infinity keeps the container as an exec host) + OS cron running docker exec <container> swamp workflow run <name> hourly — the CLI path handles the same repo fine (same pattern as the #914 workaround).

02Bog Flow

Shipped

8/7/2026, 10:07:07 PM

Click a lifecycle step above to view its details.

03Sludge Pulse

stack72 assigned stack728/7/2026, 6:14:18 PM

magistr commented 8/7/2026, 8:28:08 PM

Extra environment detail from the affected host, in case it helps narrow the OOM analysis: neither serve container has a memory limit, so the ~4 GB ceiling in the traces is not a cgroup cap.

Measured on the Unraid host (Docker, cgroup v2), 2026-08-07:

container	`/sys/fs/cgroup/memory.max`
`swamp-serve`	`max`
`swamp-serve-move` (the repo in this report)	`max`

Host: MemTotal: 65726192 kB (~64 GB), 32 cores.

So the process reached 4083.0 (4086.1) MB and died with the container cgroup unlimited on a 64 GB box — the heap ceiling comes from inside the binary rather than from the container.

Also probed whether the compiled binary takes a runtime V8 flag:

$ DENO_V8_FLAGS=--bogus-flag-xyz swamp --version
swamp 20260728.192016.0-sha.f2a1f6b9
rc=0

An unrecognized V8 flag produced no complaint and no failure, i.e. DENO_V8_FLAGS does not appear to reach V8 in the standalone build.

stack72 commented 8/7/2026, 8:56:06 PM

@magistr — PR #2097 fixes the workflow history get OOM. The root cause was findLatestByWorkflowId loading all 415 run files as full aggregates just to return the most recent one — it now uses the runs index to load a single file.

For the swamp serve boot OOM, we haven't been able to identify the root cause from the code. The PR adds diagnostic log lines to every boot phase after "Policy snapshot loaded", so the next crash will show which phase was executing when it died.

Could you redeploy with the next release and share the serve boot logs from the crash directly with @stack72? We'll get this taken care of.

stack72 commented 8/7/2026, 10:07:33 PM

Thanks @magistr for reporting this! The fix has been merged and a release is on its way. We appreciate your contribution to swamp.

magistr commented 8/7/2026, 10:35:11 PM

@stack72 — redeployed on 20260807.203020.0-sha.02bf006e. Splitting the two symptoms:

1. workflow history get — FIXED. Same repo, same 415-run history that OOMed before:

$ swamp workflow history get @magistr/move-dossier-full --repo-dir /workspace --json
rc=0 — 199909 bytes of JSON, empty stderr, 216 s wall

216 s is slow for what should be a single-run lookup, but it completes and the heap stays flat. Thanks for the quick turnaround on #2097.

2. serve boot OOM — still reproduces, and your new phase logging pins it down. Serve enabled on the same repo, 2 crash-restarts in 4 minutes, every boot identical:

2026-08-07T22:30:12.962Z [INF] serve: Policy snapshot loaded (reload mode: "manual")
2026-08-07T22:30:12.963Z [INF] serve: Boot: reaping stale runs via tracker
2026-08-07T22:30:12.976Z [INF] serve: Boot: reconciling workflow run state
[7:0x56546e704000]    86125 ms: Incremental Mark-Compact (reduce) 4088.1 (4093.9) -> 4087.8 (4091.2) MB, pooled: 0.0 MB, 1560.03 / 0.00 ms (+ 18.4 ms in 4 steps since start of marking, biggest step 5.0 ms, walltime since start of marking 1587 ms) (average
#
# Fatal JavaScript out of memory: Ineffective mark-compacts near heap limit
#
==== C stack trace ===============================
    swamp(+0x23efc5e) [0x56546ac90c5e]
    swamp(+0x23eec7b) [0x56546ac8fc7b]
    swamp(+0x23f0ec8) [0x56546ac91ec8]
    swamp(+0x249fb75) [0x56546ad40b75]
    swamp(+0x2656487) [0x56546aef7487]
    swamp(+0x2655d1d) [0x56546aef6d1d]
    swamp(+0x2654904) [0x56546aef5904]
    swamp(+0x25f362b) [0x56546ae9462b]
    swamp(+0x2a3f34d) [0x56546b2e034d]
    [0x14811be81fb6]

So the dying phase is Boot: reconciling workflow run state. Both new lines print, then it spends ~86 s allocating and dies at ~4088 MB. Nothing after those two lines ever appears.

Watched live during the climb: RSS 4.34 GiB at ~75 s in, CPU ~103% (one core saturated), container memory limit still unlimited.

Repo shape unchanged from the report: 415 run files / 22 MB in .swamp/workflow-runs/<id>/, ~400 dataArtifacts per run.

Back on the sleep infinity + cron bridge meanwhile, now running the new binary. Happy to run any instrumented build or dump whatever state would help.

stack72 commented 8/7/2026, 10:54:04 PM

@magistr — thanks for the detailed logs and the redeployment turnaround, that pinpointed everything we needed.

1. workflow history get (216s): That was a one-time index build — the fix in #2097 introduced a per-workflow .runs-index.json that makes subsequent lookups near-instant, but your 415 runs predated the fix so the first access had to parse all of them to build the index. Now that it's built, future workflow history get calls on that workflow will be fast. If you have other workflows with large histories that haven't been accessed yet, those will also see a one-time slow first call, then fast after that. No action needed on your end.

2. Serve boot OOM: Filed as #1558. The root cause is the same load-all-to-find-few pattern: the boot reconciliation phase loads every workflow run from the last 7 days as a full aggregate (jobs, steps, outputs, 400 data artifacts each) just to find the handful with status: running. We're going to use the runs index to identify running runs first and only load those. Fix incoming.

Improve multi-word search query handling and guidance

swamp command doesn't check cwd

swamp auth login process fails when run remotely

serve boot OOM: findAllGlobalSince loads all workflow runs as full aggregates during reconciliation

extension search: channel-filtered results carry null version fields, and multiple --channel values collapse instead of unioning

s3-datastore: localHasAllRemoteEntries uses bare rel instead of localRelPath, always returns false in namespaced mode

datastoreSetupExtension migrates .swamp/ to cache root instead of namespace directory

swamp serve writes outputs/ outside the bound namespace — silently never persisted to the remote datastore

s3-datastore: bulk pull writes to cache root instead of the namespace, wedging push and arming a full-datastore delete

git: add pull, fetch --tags, and cherry-pick methods

Workflow runs with wide dataArtifact fan-out (~400/run) OOM both workflow history get and swamp serve boot, crash-looping the scheduler; rollback doesn't help (state-triggered, not release-triggered)

Show user:<name> instead of user:<id> in grant list output

Reusable long-lived invite link: one URL to bring people into the swamp

Exclude swamp internal model types from model list output

Skill docs use invalid two-argument 'vault get' form

access token mint prints an invalid retrieval command (vault get, not vault read-secret)

serve reload and model edit timeout over --server in Kubernetes

swamp auth whoami should report paid tier and trial status

Use whoami entitlement to explain private-extension push refusals

Document --restricted-commands flag for swamp serve

vault put: pass tags through to extension provider CreateSecret/setSecret calls

Docs: update manual for 23 new --server commands

Update VaultProvider interface in @systeminit/swamp-testing to support optional tags in put()

Docs: document --grants-file flag for swamp serve

Docs: document --server support for serve reload

Docs: add --tag flag to vault put reference

Docs: update manual for --detach-runs deprecation and HA auto-detection

Document the Codex plugin in swamp docs

Add --server support to CLI commands missing it

Document the swamp Claude Code plugin in the swamp documentation

Make --server a global CLI option

Add --server support to model cancel (server-side run cancellation)

Load grants from external source on swamp serve startup

swamp vault put should support tags for secret governance

Improve hot reload of the deamon, include extension source/bundle changes

Authenticated users must re-login after swamp serve restart

Add --server support to serve reload

Harden queue claim semantics: atomic claims + status-guarded markDelivered/markFailed across app queues

Auto-detect HA mode and deprecate --detach-runs flag

version-drift check (from #236) is a raw string-inequality, false-positives on multi-model manifests and metadata-only bumps

Resend newsletter audience sync: contacts, segments, backfill, unsubscribe sync-back

Move per-login token secrets from vault to encrypted control-plane store

Docs: document macOS TLS trust limitation and SSL_CERT_FILE workaround

Validate swamp serve deployment stack on startup

Docs: serve config file (.swamp/serve.yaml) and --config flag

Serve config file — read swamp serve configuration from YAML

Deno runtime does not trust ISRG Root YR — extensions fail with valid Let's Encrypt certs

Docs: update hot-reload reference to include trust list refresh

Cross-instance client re-attach for multi-instance swamp serve

vercel codegen: inline rawSyncResult in non-unwrap sync path

vercel codegen: unused listAll import blocks extension publish for 4 models

vercel codegen: add round-trip idempotency test for the pipeline

Operatives button on Collectives page does nothing

Track active runs in the control-plane store for cross-instance visibility

validateGlobalArgInputRefs checks step-level inputs but step schema rejects 'inputs' key

Docs: document continuous reconciliation and multi-instance serve flags

Docs: update workflow guard skip output to include guardExpression and guardResult

Make it aware hot reload on the deamon does not take into account trust list changes, or do take into account the trust list for the hot reload.

workflow validate: new 'GlobalArgument input references' check resolves model-definition ${{ inputs.* }} against workflow inputs, false-failing steps that supply the input

Continuous reconciliation for multi-instance swamp serve

Split the score-read circuit breaker so leaderboard pressure can't blank profile scores

oauth_sign_in telemetry records provider as the literal ":id" for every event

Ghost profile bypasses the score-read seam and 500s on a slow ledger

Profile score section blocks the render on a ClickHouse read

Council: set the scoring balance — command tiers, farm-decay parameters, cross-category ratio

Scoring coverage gaps: instrument drop-off, score ripples/mentions, reward invite-converts

Push down the two cheap leaderboard scans: ghost-gate dictionary and eventCounts owner filter

Materialize an active-days-by-id table for the streak board

score-rollup-projector: O(all-time ledger) closure + per-pod fan-out saturates prod ClickHouse CPU

Startup cache hydration for swamp serve instance replacement

Output a log line for steps skipped due to guards

extension quality scorer misreads a template-literal import specifier in generated code as an unresolvable bare import, hard-erroring the whole extension as UNSCORABLE

Add @swamp/kubernetes/statefulset model type

Docs: access token commands reference says plaintext is 'shown once'

Document --restricted-model-types flag for swamp serve

swamp auth server-login ignores SWAMP_SERVE_URL

s3-datastore extension: pullChanged writes namespace-stripped paths to local cache

serve oauth: an unrecognized device-token error kills the whole serve process instead of the bootstrap

s3-datastore: bidirectional sync pull writes namespace-stripped copies to cache root, wedging push with no supported recovery

CLI reports 'HTTP/2 not supported by this client' when the server actually returned 401 — masks auth failures as a transport error