Every swamp auth server-login against an OAuth-mode server mints a new oauth-<random> server token with a 30-day expiry (DEFAULT_DURATION_MS). Old tokens are never cleaned up, even after they expire. Over time this accumulates unbounded state across four locations:

Definition YAML in .swamp/auto-definitions/swamp/server-token/<id>.yaml
Token data record in the unified data repo (e.g. GCS)
Token plaintext secret in the control plane vault (_token-secrets)
OAuth access token in the control plane store (oauth-access-token-<name>)

A team of 10 people logging in once a day generates 300 stale token records per month, none of which are ever removed.

Proposed Solution

Add a periodic GC sweep to swamp serve that cleans up expired (and revoked) server tokens. The sweep should:

Scan all swamp/server-token definitions
For each token where state === "expired" or state === "revoked" (or expiresAt has passed), delete:
- The token secret from the control plane vault (_token-secrets)
- The OAuth access token from the control plane store (oauth-access-token-<name>)
- The token data record from the unified data repo
- The definition YAML from auto-definitions
Run on a configurable interval (e.g. --token-gc-interval, default hourly)
Run once at startup after token secret migration completes
Log the count of cleaned-up tokens at info level

Considerations

Grace period — don't GC tokens immediately on expiry. A short grace period (e.g. 1 hour past expiresAt) avoids racing with a token that's about to be refreshed or a clock skew edge case.
Active sessions — an expired token may still have active WebSocket connections that authenticated before expiry. GC should not force-disconnect these; the session timeout handles that independently.
Revoked tokens — these can be GC'd immediately since revocation is a deliberate admin action.
swamp access token list — should continue to show recently expired/revoked tokens for auditability. Consider a --include-gc'd flag or a retention window for the list view.

Alternatives

CLI command (swamp access token gc) — manual cleanup, useful but doesn't solve the unbounded growth for unattended servers
TTL on the control plane store — if the datastore supports object TTL (e.g. GCS lifecycle rules), expired secrets could be auto-deleted at the storage layer, but this doesn't clean up definitions or data records

02Bog Flow

Open

8/12/2026, 12:07:24 AM

No activity in this phase yet.

03Sludge Pulse

Support --grants-dir for loading grants from a directory

migrate-index still drops shallow-path entries on 2026.08.07.1 — #1584 marked shipped but no released version carries the fix (self-contained repro)

Server token definitions and encryption keys are not durable across pod restarts

data.latest() is memoized per run, contradicting the documented "sees current-run data" contract

Model lookup misreports EMFILE (too many open files) as a broken definition file and dumps a raw stacktrace

Lab Insights not publicly available

Client shows 'Authentication failed' when actually rate-limited

Workers retry forever on permanent auth failures instead of backing off

Docs: add swamp access group list-idp to access commands reference

serve telemetry identity warnings are diagnostic but not actionable — say how to fix them

swamp quest: drop the heartbeat field the server no longer sends

--grant-reload auto does not detect filesystem changes — blind to ConfigMap updates

WebSocket auth rate limiter blocks legitimate users when sharing IP with a failing client

Add native Bedrock Knowledge Base Retrieve support

GC expired OAuth server tokens and control plane secrets

migrate-index silently drops entries with shallow paths — all of workflows-evaluated/ and any root-level file

Cluster health snapshots with SSE streaming for swamp serve

Add swamp access group list-idp to enumerate IdP groups seen by the server

Grants directory loader skips symlinks — breaks Kubernetes ConfigMap mounts

Daemon telemetry flush: stop() can skip its final flush when a timer drain is already in flight

Docs: `swamp compile` subcommand doesn't exist

Guard the projector's full rewrites against the fail-open lease (ADV-6 from #1575/#1576)

Projector write amplification: board_totals/owner_devices full rewrites are owners × runs/day

Profile candidateIds prune costs a full ledger scan to build itself — 42.88M rows for one keyed read

Leaderboard phase 3: owner-grain board_totals so a cache miss is survivable at 10B

swamp serve is telemetry-dark: scheduled workflow runs never reach /ingest, and the daemon never flushes

CLI has never sent insert_id: retried telemetry batches are double-counted into cli_daily, inflating XP in an append-only ledger

Catalog backfill reconciles against the local cache, not the datastore

Extension bundle cache not invalidated when an imported sibling module changes (entry-file fingerprint only)

swamp serve: data.query fails permanently with "UNIQUE constraint failed: catalog.*" — one duplicate catalog 5-tuple aborts every catalog rebuild

dailyHistory range pushdown — deferred from #1576, and the measurement says why

score-rollup projector runs N× per cadence — the run gate is per-replica memory, not shared state

Docs: manual never mentions telemetry, and the serve/scheduling pages imply automated runs are visible when they are not

Explore: CLI binary update integrity lacks independent verification

Leaderboard phase 2: take score_grants off the board read path — read today from score_daily

Explore: reconcile quality rubric's dormant provenance factor

Explore: pull-time signature verification for extensions

Explore: transparency-log integration (Rekor) for extension signatures

Explore: publish-time artifact signing for extensions (cosign keyless)

Profile day drill-down returns 0 entries at the 5s ClickHouse ceiling for every operative

A ledger read failure still renders as 'no activity': loadProfileScore swallows to an empty standing behind HTTP 200

reconcile_from_disk_bench.ts fails deno check (stale DenoRuntime stub)

Improve multi-word search query handling and guidance

Retire the genesis_public flag and delete the dead cutover transcription scaffolding

Leaderboard phase 1: cache the board read and bust it on tier crossings (no schema change)

Catalog backfill deletes rows for any model whose data names are numeric

Docs: document catalog completeness check and --repair in doctor reference

Leaderboard board windows re-anchor on wall-clock while the rest of the view is frozen at asOf

Decision: move per-event grant MVs to day-grain — ledger is on track for ~1.5B score_grants / ~4B quest_grants

Leaderboard phase 5: score-rollup-projector write cost at 10B — ~430M rows scanned per 60s tick

Epic: /leaderboard and /u/{username} do not survive 10B events — measured audit + phased plan

Datastore-managed configuration for multi-instance serve deployments

Ghosts on the leaderboard 500

swamp command doesn't check cwd

swamp auth login process fails when run remotely

serve boot OOM: findAllGlobalSince loads all workflow runs as full aggregates during reconciliation

extension search: channel-filtered results carry null version fields, and multiple --channel values collapse instead of unioning

s3-datastore: localHasAllRemoteEntries uses bare rel instead of localRelPath, always returns false in namespaced mode

datastoreSetupExtension migrates .swamp/ to cache root instead of namespace directory

swamp serve writes outputs/ outside the bound namespace — silently never persisted to the remote datastore

s3-datastore: bulk pull writes to cache root instead of the namespace, wedging push and arming a full-datastore delete

git: add pull, fetch --tags, and cherry-pick methods

Workflow runs with wide dataArtifact fan-out (~400/run) OOM both workflow history get and swamp serve boot, crash-looping the scheduler; rollback doesn't help (state-triggered, not release-triggered)

Show user:<name> instead of user:<id> in grant list output

Reusable long-lived invite link: one URL to bring people into the swamp

Exclude swamp internal model types from model list output

Skill docs use invalid two-argument 'vault get' form

access token mint prints an invalid retrieval command (vault get, not vault read-secret)

serve reload and model edit timeout over --server in Kubernetes

swamp auth whoami should report paid tier and trial status

Use whoami entitlement to explain private-extension push refusals

Document --restricted-commands flag for swamp serve

vault put: pass tags through to extension provider CreateSecret/setSecret calls

Docs: update manual for 23 new --server commands

Update VaultProvider interface in @systeminit/swamp-testing to support optional tags in put()

Docs: document --grants-file flag for swamp serve

Docs: document --server support for serve reload

Docs: add --tag flag to vault put reference

Docs: update manual for --detach-runs deprecation and HA auto-detection

Document the Codex plugin in swamp docs