HttpTelemetrySender.sendBatch has never sent an insert_id. swamp-club's ingest queue carries a unique index on that field specifically to deduplicate a re-POST, and its own comment names swamp as the emitter that gets this wrong. Every retry of an already-accepted batch has therefore been landing as a second queue doc and double-counting into cli_daily — which feeds the activity projector and the score ledger.

The forward fix ships in swamp-club/swamp#2117 (a one-line change, found while fixing swamp-club#1591). This issue is about the inflation already banked, which that PR does not and cannot address.

Evidence

services/telemetry/lib/db.ts:78-81 (swamp-club), verbatim:

Idempotent ingest for emitters that supply a client-side insert_id (reliableTrack does; legacy/CLI emitters get a server-generated one): a retried POST re-sends the accepted key, and without this it becomes a new queue doc in a LATER batch, duplicated into every consumer — the Mongo consumers dedup via their *_processed_inserts collections, but ClickHouse would double-count cli_daily (permanent XP inflation; the per-batch insert_deduplication_token can't see across batches).

And toEventDoc (services/telemetry/lib/schema.ts:255):

insert_id: input.insert_id ?? crypto.randomUUID(),

The CLI sender sent only { event, distinct_id, properties }, so it took the crypto.randomUUID() branch on every POST — including retries of a batch the server had already accepted.

When this fires

TelemetryService.doFlush only marks entries flushed after a successful send. Any path where the server accepts a batch but the client does not learn it re-sends the whole batch on the next invocation:

A 202 lost to a network drop, or to the sender's own 5s AbortSignal.timeout firing after the server committed.
The 2s flush timeout in the CLI teardown (cli/mod.ts:1619) expiring mid-request.
Any process death between the server's commit and the local markFlushed.

Batches are up to 25 entries, so one lost ack inflates up to 25 invocations at once.

Impact

swamp.cli_daily_mv counts every cli_invocation row with result.status = 'success', and the activity projector values that count with no dedup of its own. Duplicates therefore raise cnt, which raises granted units. The ledger is explicitly append-only — activity-asof-projector.sql states points already banked are never clawed back — so this is not self-correcting.

Magnitude is unknown and may well be negligible. That is the point of filing: it has never been measured.

Suggested next step

Duplicates are retroactively detectable, because the entry's own UUID is carried in the payload as properties.id and is stable across retries even though insert_id was not. Roughly:

SELECT count() AS dup_groups, sum(n - 1) AS surplus_rows
FROM (
  SELECT JSONExtractString(properties, 'id') AS entry_id, count() AS n
  FROM swamp.events
  WHERE event = 'cli_invocation'
  GROUP BY entry_id
  HAVING n > 1
);

That gives the surplus row count directly. If it is material, decide deliberately whether to correct it or accept it and note it — a silent correction to an append-only ledger is worse than a documented one.

Environment

swamp CLI, all versions prior to swamp-club/swamp#2117
Applies to the interactive CLI path, not just daemons — this predates and is independent of swamp-club#1591

swamp-club/swamp#2117 — sends insert_id: entry.id, stopping further inflation
#1591 — serve telemetry gap, where this was discovered
#1254 — telemetry stats consumer dedup ordering (different layer, same field)
#820 — telemetry watcher replica coordination (shipped)

02Bog Flow

Open

8/11/2026, 4:02:45 PM

Click a lifecycle step above to view its details.

03Sludge Pulse

stack72 assigned stack728/11/2026, 8:02:54 PM

stack72 unassigned stack728/11/2026, 9:33:06 PM

Server token definitions are not durable across pod restarts

WebSocket auth rate limiter blocks legitimate users when sharing IP with a failing client

Add native Bedrock Knowledge Base Retrieve support

GC expired OAuth server tokens and control plane secrets

migrate-index silently drops entries with shallow paths — all of workflows-evaluated/ and any root-level file

Cluster health snapshots with SSE streaming for swamp serve

Add swamp access group list-idp to enumerate IdP groups seen by the server

Docs: add swamp access group list-idp to access commands reference

Grants directory loader skips symlinks — breaks Kubernetes ConfigMap mounts

--grant-reload auto does not detect filesystem changes — blind to ConfigMap updates

Daemon telemetry flush: stop() can skip its final flush when a timer drain is already in flight

Support --grants-dir for loading grants from a directory

Docs: `swamp compile` subcommand doesn't exist

Guard the projector's full rewrites against the fail-open lease (ADV-6 from #1575/#1576)

Projector write amplification: board_totals/owner_devices full rewrites are owners × runs/day

Profile candidateIds prune costs a full ledger scan to build itself — 42.88M rows for one keyed read

Leaderboard phase 3: owner-grain board_totals so a cache miss is survivable at 10B

serve telemetry identity warnings are diagnostic but not actionable — say how to fix them

swamp serve is telemetry-dark: scheduled workflow runs never reach /ingest, and the daemon never flushes

CLI has never sent insert_id: retried telemetry batches are double-counted into cli_daily, inflating XP in an append-only ledger

Catalog backfill reconciles against the local cache, not the datastore

Extension bundle cache not invalidated when an imported sibling module changes (entry-file fingerprint only)

swamp serve: data.query fails permanently with "UNIQUE constraint failed: catalog.*" — one duplicate catalog 5-tuple aborts every catalog rebuild

dailyHistory range pushdown — deferred from #1576, and the measurement says why

score-rollup projector runs N× per cadence — the run gate is per-replica memory, not shared state

Docs: manual never mentions telemetry, and the serve/scheduling pages imply automated runs are visible when they are not

Explore: CLI binary update integrity lacks independent verification

Leaderboard phase 2: take score_grants off the board read path — read today from score_daily

Explore: reconcile quality rubric's dormant provenance factor

Explore: pull-time signature verification for extensions

Explore: transparency-log integration (Rekor) for extension signatures

Explore: publish-time artifact signing for extensions (cosign keyless)

Profile day drill-down returns 0 entries at the 5s ClickHouse ceiling for every operative

A ledger read failure still renders as 'no activity': loadProfileScore swallows to an empty standing behind HTTP 200

swamp quest: drop the heartbeat field the server no longer sends

reconcile_from_disk_bench.ts fails deno check (stale DenoRuntime stub)

Improve multi-word search query handling and guidance

Retire the genesis_public flag and delete the dead cutover transcription scaffolding

Leaderboard phase 1: cache the board read and bust it on tier crossings (no schema change)

Catalog backfill deletes rows for any model whose data names are numeric

Docs: document catalog completeness check and --repair in doctor reference

Leaderboard board windows re-anchor on wall-clock while the rest of the view is frozen at asOf

Decision: move per-event grant MVs to day-grain — ledger is on track for ~1.5B score_grants / ~4B quest_grants

Leaderboard phase 5: score-rollup-projector write cost at 10B — ~430M rows scanned per 60s tick

Epic: /leaderboard and /u/{username} do not survive 10B events — measured audit + phased plan

Datastore-managed configuration for multi-instance serve deployments

Ghosts on the leaderboard 500

swamp command doesn't check cwd

swamp auth login process fails when run remotely

serve boot OOM: findAllGlobalSince loads all workflow runs as full aggregates during reconciliation

extension search: channel-filtered results carry null version fields, and multiple --channel values collapse instead of unioning

s3-datastore: localHasAllRemoteEntries uses bare rel instead of localRelPath, always returns false in namespaced mode

datastoreSetupExtension migrates .swamp/ to cache root instead of namespace directory

swamp serve writes outputs/ outside the bound namespace — silently never persisted to the remote datastore

s3-datastore: bulk pull writes to cache root instead of the namespace, wedging push and arming a full-datastore delete

git: add pull, fetch --tags, and cherry-pick methods

Workflow runs with wide dataArtifact fan-out (~400/run) OOM both workflow history get and swamp serve boot, crash-looping the scheduler; rollback doesn't help (state-triggered, not release-triggered)

Show user:<name> instead of user:<id> in grant list output

Reusable long-lived invite link: one URL to bring people into the swamp

Exclude swamp internal model types from model list output

Skill docs use invalid two-argument 'vault get' form

access token mint prints an invalid retrieval command (vault get, not vault read-secret)

serve reload and model edit timeout over --server in Kubernetes

swamp auth whoami should report paid tier and trial status

Use whoami entitlement to explain private-extension push refusals

Document --restricted-commands flag for swamp serve

vault put: pass tags through to extension provider CreateSecret/setSecret calls

Docs: update manual for 23 new --server commands

Update VaultProvider interface in @systeminit/swamp-testing to support optional tags in put()

Docs: document --grants-file flag for swamp serve

Docs: document --server support for serve reload

Docs: add --tag flag to vault put reference

Docs: update manual for --detach-runs deprecation and HA auto-detection

Document the Codex plugin in swamp docs

Add --server support to CLI commands missing it

Document the swamp Claude Code plugin in the swamp documentation

Make --server a global CLI option

Add --server support to model cancel (server-side run cancellation)

Load grants from external source on swamp serve startup

swamp vault put should support tags for secret governance