.swamp/_control/active-runs/ accumulates empty directories that are never removed after runs complete. Observed 6 stale directories, the oldest two days old, with run doctor reporting clean throughout.

Less severe than the pending-run leak (they don't replay on restart), but the same missing-cleanup shape — the ActiveRunHook.delete path likely has a similar issue to the pending-run cleanup.

Observed during investigation of #1627 (pending-run leak). The ActiveRunHook interface in scheduled_execution.ts also uses void return types with fire-and-forget controlPlaneStore.delete() calls.

Expected behavior

Empty active-runs/ directories should be cleaned up when the run completes or on boot reconciliation.

02Bog Flow

Triaged

8/12/2026, 10:42:58 PM

Click a lifecycle step above to view its details.

03Sludge Pulse

stack72 assigned stack728/12/2026, 10:39:35 PM

Failing on claude validate

GC expired OAuth server tokens and control plane secrets

Client shows 'Authentication failed' when actually rate-limited

serve: cron ticks leak pending-run records, replaying every past tick on restart

Workers retry forever on permanent auth failures instead of backing off

swamp serve dies of descriptor exhaustion on an ordinary workload, having warned about it at startup

Remote execution has no working network configuration: off-loopback serve requires TLS, but worker connect cannot connect over TLS

serve: stale empty directories in active-runs/ never cleaned up

Support --grants-dir for loading grants from a directory

migrate-index still drops shallow-path entries on 2026.08.07.1 — #1584 marked shipped but no released version carries the fix (self-contained repro)

Server token definitions and encryption keys are not durable across pod restarts

data.latest() is memoized per run, contradicting the documented "sees current-run data" contract

Model lookup misreports EMFILE (too many open files) as a broken definition file and dumps a raw stacktrace

Lab Insights not publicly available

Docs: add swamp access group list-idp to access commands reference

serve telemetry identity warnings are diagnostic but not actionable — say how to fix them

swamp quest: drop the heartbeat field the server no longer sends

--grant-reload auto does not detect filesystem changes — blind to ConfigMap updates

WebSocket auth rate limiter blocks legitimate users when sharing IP with a failing client

Add native Bedrock Knowledge Base Retrieve support

migrate-index silently drops entries with shallow paths — all of workflows-evaluated/ and any root-level file

Cluster health snapshots with SSE streaming for swamp serve

Add swamp access group list-idp to enumerate IdP groups seen by the server

Grants directory loader skips symlinks — breaks Kubernetes ConfigMap mounts

Daemon telemetry flush: stop() can skip its final flush when a timer drain is already in flight

Docs: `swamp compile` subcommand doesn't exist

Guard the projector's full rewrites against the fail-open lease (ADV-6 from #1575/#1576)

Projector write amplification: board_totals/owner_devices full rewrites are owners × runs/day

Profile candidateIds prune costs a full ledger scan to build itself — 42.88M rows for one keyed read

Leaderboard phase 3: owner-grain board_totals so a cache miss is survivable at 10B

swamp serve is telemetry-dark: scheduled workflow runs never reach /ingest, and the daemon never flushes

CLI has never sent insert_id: retried telemetry batches are double-counted into cli_daily, inflating XP in an append-only ledger

Catalog backfill reconciles against the local cache, not the datastore

Extension bundle cache not invalidated when an imported sibling module changes (entry-file fingerprint only)

swamp serve: data.query fails permanently with "UNIQUE constraint failed: catalog.*" — one duplicate catalog 5-tuple aborts every catalog rebuild

dailyHistory range pushdown — deferred from #1576, and the measurement says why

score-rollup projector runs N× per cadence — the run gate is per-replica memory, not shared state

Docs: manual never mentions telemetry, and the serve/scheduling pages imply automated runs are visible when they are not

Explore: CLI binary update integrity lacks independent verification

Leaderboard phase 2: take score_grants off the board read path — read today from score_daily

Explore: reconcile quality rubric's dormant provenance factor

Explore: pull-time signature verification for extensions

Explore: transparency-log integration (Rekor) for extension signatures

Explore: publish-time artifact signing for extensions (cosign keyless)

Profile day drill-down returns 0 entries at the 5s ClickHouse ceiling for every operative

A ledger read failure still renders as 'no activity': loadProfileScore swallows to an empty standing behind HTTP 200

reconcile_from_disk_bench.ts fails deno check (stale DenoRuntime stub)

Improve multi-word search query handling and guidance

Retire the genesis_public flag and delete the dead cutover transcription scaffolding

Leaderboard phase 1: cache the board read and bust it on tier crossings (no schema change)

Catalog backfill deletes rows for any model whose data names are numeric

Docs: document catalog completeness check and --repair in doctor reference

Leaderboard board windows re-anchor on wall-clock while the rest of the view is frozen at asOf

Decision: move per-event grant MVs to day-grain — ledger is on track for ~1.5B score_grants / ~4B quest_grants

Leaderboard phase 5: score-rollup-projector write cost at 10B — ~430M rows scanned per 60s tick

Epic: /leaderboard and /u/{username} do not survive 10B events — measured audit + phased plan

Datastore-managed configuration for multi-instance serve deployments

Ghosts on the leaderboard 500

swamp command doesn't check cwd

swamp auth login process fails when run remotely

serve boot OOM: findAllGlobalSince loads all workflow runs as full aggregates during reconciliation

extension search: channel-filtered results carry null version fields, and multiple --channel values collapse instead of unioning

s3-datastore: localHasAllRemoteEntries uses bare rel instead of localRelPath, always returns false in namespaced mode

datastoreSetupExtension migrates .swamp/ to cache root instead of namespace directory

swamp serve writes outputs/ outside the bound namespace — silently never persisted to the remote datastore

s3-datastore: bulk pull writes to cache root instead of the namespace, wedging push and arming a full-datastore delete

git: add pull, fetch --tags, and cherry-pick methods

Workflow runs with wide dataArtifact fan-out (~400/run) OOM both workflow history get and swamp serve boot, crash-looping the scheduler; rollback doesn't help (state-triggered, not release-triggered)

Show user:<name> instead of user:<id> in grant list output

Reusable long-lived invite link: one URL to bring people into the swamp

Exclude swamp internal model types from model list output

Skill docs use invalid two-argument 'vault get' form

access token mint prints an invalid retrieval command (vault get, not vault read-secret)

serve reload and model edit timeout over --server in Kubernetes

swamp auth whoami should report paid tier and trial status

Use whoami entitlement to explain private-extension push refusals

Document --restricted-commands flag for swamp serve

vault put: pass tags through to extension provider CreateSecret/setSecret calls

Docs: update manual for 23 new --server commands

Update VaultProvider interface in @systeminit/swamp-testing to support optional tags in put()