When swamp serve mints a server token (via OAuth login or access token mint), the token state is split across three stores with different durability:

Definition YAML → local filesystem (.swamp/auto-definitions/swamp/server-token/<id>.yaml)
Token data record → shared datastore (GCS)
Token secret → shared control plane vault (_token-secrets in GCS)

If the pod is replaced before the definition file syncs to GCS — or if sync never happens — the definition is lost. The secret and data remain in GCS but can never be used because authenticateServerToken runs modelMethodRun which requires the definition to exist. The token is permanently broken.

Encryption key durability

The control plane vault encrypts all token secrets with an AES-GCM key stored at token-secrets/encryption-key in the control plane store. In a shared store (GCS), the key is written once via putIfAbsent and shared across pods. However, if the key is lost — GCS bucket wipe, namespace/prefix change, fallback to local filesystem — a new key is generated and all existing token secrets become undecryptable, even if the definitions and data survive.

Failure modes:

Namespace/prefix change between deployments — old key is invisible under the new prefix, new key generated, all existing tokens broken
putIfAbsent bug in the datastore extension — if the implementation doesn't use proper preconditions, concurrent pods could write different keys
Fallback to local filesystem — if the remote control plane store isn't available at startup, FileSystemControlPlaneStore is used and the key is ephemeral

Diagnostic: check server startup logs for "Generated new token encryption key" (new key — potential problem) vs "Loaded existing token encryption key" (existing key found — correct).

Impact

Users or workers with the affected token get "Authentication failed" indefinitely
The failing client retries, which can trigger the per-IP rate limiter and block other legitimate clients on the same IP (#1615)
Orphaned secrets and data accumulate in the control plane store with no corresponding definition, invisible to any scan or GC (#1614)
No error or warning at mint time — the operation appears to succeed

Observed in production

A worker token platform-fleet-gke-svc was minted on a previous pod incarnation. After pod replacement, the definition was lost. The worker retried auth every second, producing "Model not found: platform-fleet-gke-svc" on every attempt, burning through the rate limit budget and blocking all other clients on the same cluster IP.

Proposed Fix

Definition durability

The mint operation should be atomic — the definition must land in the durable shared store before the token is returned to the client. Options:

Store definitions in the shared datastore — write the definition to GCS alongside the data and secret, not to local YAML. The definition repo should be able to read from the shared store.
Sync before returning — ensure syncService.pushChanged() completes and includes the definition file before returning the token to the caller. Today the definition is written via a local YamlDefinitionRepository that may not be covered by the sync.
Verify round-trip — after mint, verify the definition can be read back via the same code path that authenticateServerToken uses.

Encryption key durability

Log a warning at startup if a new encryption key is generated on a server with a remote control plane store — this likely means the old key was lost
Consider storing the encryption key path outside the namespaced prefix so it survives namespace changes
Verify putIfAbsent implementations in datastore extensions use proper conditional writes (GCS: ifGenerationMatch=0, S3: If-None-Match)

Orphan cleanup

A sweep at startup should detect orphaned secrets in _token-secrets that have no corresponding definition, log them, and optionally clean them up.

02Bog Flow

In Progress

8/12/2026, 4:20:44 PM

Click a lifecycle step above to view its details.

03Sludge Pulse

stack72 assigned stack728/12/2026, 3:15:17 PM

migrate-index still drops shallow-path entries on 2026.08.07.1 — #1584 marked shipped but no released version carries the fix (self-contained repro)

data.latest() is memoized per run, contradicting the documented "sees current-run data" contract

Model lookup misreports EMFILE (too many open files) as a broken definition file and dumps a raw stacktrace

Lab Insights not publicly available

Client shows 'Authentication failed' when actually rate-limited

Workers retry forever on permanent auth failures instead of backing off

Docs: add swamp access group list-idp to access commands reference

serve telemetry identity warnings are diagnostic but not actionable — say how to fix them

swamp quest: drop the heartbeat field the server no longer sends

--grant-reload auto does not detect filesystem changes — blind to ConfigMap updates

Server token definitions and encryption keys are not durable across pod restarts

WebSocket auth rate limiter blocks legitimate users when sharing IP with a failing client

Add native Bedrock Knowledge Base Retrieve support

GC expired OAuth server tokens and control plane secrets

migrate-index silently drops entries with shallow paths — all of workflows-evaluated/ and any root-level file

Cluster health snapshots with SSE streaming for swamp serve

Add swamp access group list-idp to enumerate IdP groups seen by the server

Grants directory loader skips symlinks — breaks Kubernetes ConfigMap mounts

Daemon telemetry flush: stop() can skip its final flush when a timer drain is already in flight

Support --grants-dir for loading grants from a directory

Docs: `swamp compile` subcommand doesn't exist

Guard the projector's full rewrites against the fail-open lease (ADV-6 from #1575/#1576)

Projector write amplification: board_totals/owner_devices full rewrites are owners × runs/day

Profile candidateIds prune costs a full ledger scan to build itself — 42.88M rows for one keyed read

Leaderboard phase 3: owner-grain board_totals so a cache miss is survivable at 10B

swamp serve is telemetry-dark: scheduled workflow runs never reach /ingest, and the daemon never flushes

CLI has never sent insert_id: retried telemetry batches are double-counted into cli_daily, inflating XP in an append-only ledger

Catalog backfill reconciles against the local cache, not the datastore

Extension bundle cache not invalidated when an imported sibling module changes (entry-file fingerprint only)

swamp serve: data.query fails permanently with "UNIQUE constraint failed: catalog.*" — one duplicate catalog 5-tuple aborts every catalog rebuild

dailyHistory range pushdown — deferred from #1576, and the measurement says why

score-rollup projector runs N× per cadence — the run gate is per-replica memory, not shared state

Docs: manual never mentions telemetry, and the serve/scheduling pages imply automated runs are visible when they are not

Explore: CLI binary update integrity lacks independent verification

Leaderboard phase 2: take score_grants off the board read path — read today from score_daily

Explore: reconcile quality rubric's dormant provenance factor

Explore: pull-time signature verification for extensions

Explore: transparency-log integration (Rekor) for extension signatures

Explore: publish-time artifact signing for extensions (cosign keyless)

Profile day drill-down returns 0 entries at the 5s ClickHouse ceiling for every operative

A ledger read failure still renders as 'no activity': loadProfileScore swallows to an empty standing behind HTTP 200

reconcile_from_disk_bench.ts fails deno check (stale DenoRuntime stub)

Improve multi-word search query handling and guidance

Retire the genesis_public flag and delete the dead cutover transcription scaffolding

Leaderboard phase 1: cache the board read and bust it on tier crossings (no schema change)

Catalog backfill deletes rows for any model whose data names are numeric

Docs: document catalog completeness check and --repair in doctor reference

Leaderboard board windows re-anchor on wall-clock while the rest of the view is frozen at asOf

Decision: move per-event grant MVs to day-grain — ledger is on track for ~1.5B score_grants / ~4B quest_grants

Leaderboard phase 5: score-rollup-projector write cost at 10B — ~430M rows scanned per 60s tick

Epic: /leaderboard and /u/{username} do not survive 10B events — measured audit + phased plan

Datastore-managed configuration for multi-instance serve deployments

Ghosts on the leaderboard 500

swamp command doesn't check cwd

swamp auth login process fails when run remotely

serve boot OOM: findAllGlobalSince loads all workflow runs as full aggregates during reconciliation

extension search: channel-filtered results carry null version fields, and multiple --channel values collapse instead of unioning

s3-datastore: localHasAllRemoteEntries uses bare rel instead of localRelPath, always returns false in namespaced mode

datastoreSetupExtension migrates .swamp/ to cache root instead of namespace directory

swamp serve writes outputs/ outside the bound namespace — silently never persisted to the remote datastore

s3-datastore: bulk pull writes to cache root instead of the namespace, wedging push and arming a full-datastore delete

git: add pull, fetch --tags, and cherry-pick methods

Workflow runs with wide dataArtifact fan-out (~400/run) OOM both workflow history get and swamp serve boot, crash-looping the scheduler; rollback doesn't help (state-triggered, not release-triggered)

Show user:<name> instead of user:<id> in grant list output

Reusable long-lived invite link: one URL to bring people into the swamp

Exclude swamp internal model types from model list output

Skill docs use invalid two-argument 'vault get' form

access token mint prints an invalid retrieval command (vault get, not vault read-secret)

serve reload and model edit timeout over --server in Kubernetes

swamp auth whoami should report paid tier and trial status

Use whoami entitlement to explain private-extension push refusals

Document --restricted-commands flag for swamp serve

vault put: pass tags through to extension provider CreateSecret/setSecret calls

Docs: update manual for 23 new --server commands

Update VaultProvider interface in @systeminit/swamp-testing to support optional tags in put()

Docs: document --grants-file flag for swamp serve

Docs: document --server support for serve reload

Docs: add --tag flag to vault put reference

Docs: update manual for --detach-runs deprecation and HA auto-detection

Document the Codex plugin in swamp docs