PR #2139 (swamp-club#1613) adds three new authenticated HTTP endpoints to swamp serve for per-node health monitoring. The manual at content/manual/reference/swamp-serve/rest-api.md needs to document these.

Endpoints to Document

GET /api/v1/health — Health Snapshot

Returns a comprehensive point-in-time JSON snapshot of node health:

Instance identity: instanceId, deployment mode, uptime, readiness state
Active runs: run IDs, kinds (workflow-run/method-run), resource names, durations, principals
Throughput metrics: sliding-window (5 min) counters — completions, failures, cancellations, throughput/min, latency percentiles (P50/P95/P99)
Workers: name, status (idle/busy/unverified/draining), connected state, capacity, active dispatch count, platform
Scheduling: enabled state, schedule entries with cron expressions and next run times
Webhooks: registered routes, target workflows, verification schemes
Component health: datastore (reachability + latency), vault (availability) — each reports healthy/unhealthy, message, latency, details

Auth: Bearer [REDACTED-SECRET-1] + admin authorization (same as /api/v1/cancel).

GET /api/v1/health/stream — SSE Health Stream

Server-Sent Events stream that pushes health snapshots at a configurable interval.

Query param ?interval=<ms> controls push frequency (default 5000ms, min 1000, max 60000)
Supports Last-Event-ID header for resumable connections
SSE format: id: <n>\nevent: health\ndata: <json>\n\n
Sends initial snapshot immediately on connect, then at interval
Includes X-Accel-Buffering: no header for nginx reverse proxy compatibility

Auth: Same bearer token + admin auth.

GET /internal/runs — Full Run History

Returns all run records from the SQLite-backed RunTrackerStore, including completed, failed, and cancelled runs with full metadata.

Disabled by default — requires --enable-internal-api flag (or enable-internal-api: true in serve config, or SWAMP_ENABLE_INTERNAL_API=true env var)
Returns 404 when the flag is not set

Auth: Same bearer token + admin auth when enabled.

Authentication Setup

Document how to obtain and use a token:

# 1. Mint a server token
swamp access token mint health-monitor --principal user:monitoring

# 2. Grant admin access
swamp access grant create --subject user:monitoring --allow admin --on "access:*"

# 3. Use it
curl -H "Authorization: Bearer health-monitor.<secret>" https://serve:9090/api/v1/health

In OAuth mode, principals listed in --admins already have admin access — any server token minted for them works without a separate grant.

Where to Document

content/manual/reference/swamp-serve/rest-api.md — add endpoint reference
content/manual/how-to/swamp-serve/ — consider a how-to guide for setting up health monitoring dashboards

02Bog Flow

Triaged

8/14/2026, 1:46:45 AM

Click a lifecycle step above to view its details.

03Sludge Pulse

stack72 assigned stack728/14/2026, 1:45:20 AM

Docs: document trigger overrides in swamp-club manual

CLI: swamp workflow trigger set/get/remove commands

Smoke test: gated lab issue lifecycle workflow

Cluster health snapshots with SSE streaming for swamp serve

Docs: document auto-raise of open-file limit in swamp serve reference

Docs: document health monitoring endpoints for swamp serve

Quality rubric + push gate for version upgrade chain correctness

workflow resume truncates the run's log file, losing every earlier attempt's records

Add an upstream-state method: status cannot answer whether commits are pushed

swamp serve dies of descriptor exhaustion on an ordinary workload, having warned about it at startup

extension quality awards repository-verified 2/2 for a nonexistent or private GitHub repo

extension/guide.md quick reference uses -j, which no swamp command accepts

Catalog write-lock contention during extension load leaves a partial registry; auto-resolve fabricates 'local edits' and suggests a destructive --force re-pull

Add a list/discover method to the generated @swamp/aws types — no way to enumerate resources in a region

serve: stale empty directories in active-runs/ never cleaned up

Extension workflows: no way to override trigger configuration

swamp serve: webhook secrets should support @vault= resolution

WebSocket auth rate limiter blocks legitimate users when sharing IP with a failing client

Failing on claude validate

GC expired OAuth server tokens and control plane secrets

Client shows 'Authentication failed' when actually rate-limited

serve: cron ticks leak pending-run records, replaying every past tick on restart

Workers retry forever on permanent auth failures instead of backing off

Remote execution has no working network configuration: off-loopback serve requires TLS, but worker connect cannot connect over TLS

Support --grants-dir for loading grants from a directory

migrate-index still drops shallow-path entries on 2026.08.07.1 — #1584 marked shipped but no released version carries the fix (self-contained repro)

Server token definitions and encryption keys are not durable across pod restarts

data.latest() is memoized per run, contradicting the documented "sees current-run data" contract

Model lookup misreports EMFILE (too many open files) as a broken definition file and dumps a raw stacktrace

Lab Insights not publicly available

Docs: add swamp access group list-idp to access commands reference

serve telemetry identity warnings are diagnostic but not actionable — say how to fix them

swamp quest: drop the heartbeat field the server no longer sends

--grant-reload auto does not detect filesystem changes — blind to ConfigMap updates

Add native Bedrock Knowledge Base Retrieve support

migrate-index silently drops entries with shallow paths — all of workflows-evaluated/ and any root-level file

Add swamp access group list-idp to enumerate IdP groups seen by the server

Grants directory loader skips symlinks — breaks Kubernetes ConfigMap mounts

Daemon telemetry flush: stop() can skip its final flush when a timer drain is already in flight

Docs: `swamp compile` subcommand doesn't exist

Guard the projector's full rewrites against the fail-open lease (ADV-6 from #1575/#1576)

Projector write amplification: board_totals/owner_devices full rewrites are owners × runs/day

Profile candidateIds prune costs a full ledger scan to build itself — 42.88M rows for one keyed read

Leaderboard phase 3: owner-grain board_totals so a cache miss is survivable at 10B

swamp serve is telemetry-dark: scheduled workflow runs never reach /ingest, and the daemon never flushes

CLI has never sent insert_id: retried telemetry batches are double-counted into cli_daily, inflating XP in an append-only ledger

Catalog backfill reconciles against the local cache, not the datastore

Extension bundle cache not invalidated when an imported sibling module changes (entry-file fingerprint only)

swamp serve: data.query fails permanently with "UNIQUE constraint failed: catalog.*" — one duplicate catalog 5-tuple aborts every catalog rebuild

dailyHistory range pushdown — deferred from #1576, and the measurement says why

score-rollup projector runs N× per cadence — the run gate is per-replica memory, not shared state

Docs: manual never mentions telemetry, and the serve/scheduling pages imply automated runs are visible when they are not

Explore: CLI binary update integrity lacks independent verification

Leaderboard phase 2: take score_grants off the board read path — read today from score_daily

Explore: reconcile quality rubric's dormant provenance factor

Explore: pull-time signature verification for extensions

Explore: transparency-log integration (Rekor) for extension signatures

Explore: publish-time artifact signing for extensions (cosign keyless)

Profile day drill-down returns 0 entries at the 5s ClickHouse ceiling for every operative

A ledger read failure still renders as 'no activity': loadProfileScore swallows to an empty standing behind HTTP 200

reconcile_from_disk_bench.ts fails deno check (stale DenoRuntime stub)

Improve multi-word search query handling and guidance

Retire the genesis_public flag and delete the dead cutover transcription scaffolding

Leaderboard phase 1: cache the board read and bust it on tier crossings (no schema change)

Catalog backfill deletes rows for any model whose data names are numeric

Docs: document catalog completeness check and --repair in doctor reference

Leaderboard board windows re-anchor on wall-clock while the rest of the view is frozen at asOf

Decision: move per-event grant MVs to day-grain — ledger is on track for ~1.5B score_grants / ~4B quest_grants

Leaderboard phase 5: score-rollup-projector write cost at 10B — ~430M rows scanned per 60s tick

Epic: /leaderboard and /u/{username} do not survive 10B events — measured audit + phased plan

Datastore-managed configuration for multi-instance serve deployments

Ghosts on the leaderboard 500

swamp command doesn't check cwd

swamp auth login process fails when run remotely

serve boot OOM: findAllGlobalSince loads all workflow runs as full aggregates during reconciliation

extension search: channel-filtered results carry null version fields, and multiple --channel values collapse instead of unioning

s3-datastore: localHasAllRemoteEntries uses bare rel instead of localRelPath, always returns false in namespaced mode

datastoreSetupExtension migrates .swamp/ to cache root instead of namespace directory

swamp serve writes outputs/ outside the bound namespace — silently never persisted to the remote datastore

s3-datastore: bulk pull writes to cache root instead of the namespace, wedging push and arming a full-datastore delete