Operators building health dashboards for swamp serve have no single endpoint to get a comprehensive, continuously updated view of whether a node is healthy, how much work it is processing, and which dependency is responsible for degradation.

Today's /health endpoint returns unconditional ok with limited schedule and webhook configuration data. /ready is a one-shot startup latch that never reverts to not-ready even if critical components fail. UIs must combine multiple diagnostic WebSocket RPCs and polling calls, but still cannot calculate trustworthy fleet-wide throughput, latency, or dependency health.

Proposed Solution

Add three new authenticated HTTP endpoints to swamp serve, each reporting per-node health (no fleet aggregation — the UI composes the fleet view from N nodes):

1. `GET /api/v1/health` — Health Snapshot

Returns a comprehensive point-in-time JSON snapshot including:

Instance identity: instanceId, deployment mode, uptime, readiness state
Active and queued runs: run IDs, kinds (workflow-run/method-run), resource names, durations, principals
Run throughput metrics: sliding-window (5 min) counters for completions, failures, cancellations, throughput per minute, and latency percentiles (P50/P95/P99)
Workers: name, status (idle/busy/unverified/draining), connected state, capacity, active dispatch count, platform
Scheduling: enabled state, schedule entries with cron expressions, next run times, running status
Webhooks: registered routes, target workflows, verification schemes
Component health: individually checked with timeouts — datastore (reachability + latency), vault (availability), and any future dependencies. Each reports healthy/unhealthy, message, latency, and details

Requires bearer token authentication + admin authorization (same pattern as the existing /api/v1/cancel endpoint).

2. `GET /api/v1/health/stream` — SSE Health Stream

A Server-Sent Events stream that pushes health snapshots at a configurable interval:

Query param ?interval=<ms> controls push frequency (default 5000ms, min 1000, max 60000)
Supports Last-Event-ID header for resumable connections
SSE format: id: <n>\nevent: health\ndata: <json>\n\n
Sends initial snapshot immediately on connect, then at interval
Includes X-Accel-Buffering: no header for nginx reverse proxy compatibility
Same bearer token + admin auth as the snapshot endpoint

SSE is chosen over WebSocket because this is a read-only, server-to-client push — exactly what SSE was designed for. It also has a much lower integration bar for dashboard tools (curl, browser EventSource, Grafana HTTP datasource) compared to the WebSocket protocol.

3. `GET /internal/runs` — Full Run History

Returns complete run history from the RunTrackerStore (SQLite-backed), including all completed, failed, and cancelled runs with their full metadata.

Disabled by default — requires --enable-internal-api flag (or enable-internal-api: true in serve config, or SWAMP_ENABLE_INTERNAL_API=true env var)
Returns 404 when the flag is not set
Same bearer token + admin auth

This separates the "how is this node doing" aggregate view (on /api/v1/health) from the "what has this node done" detail view, keeping the health snapshot lean on busy nodes.

Existing endpoints unchanged

/health and /ready remain as lightweight, unauthenticated Kubernetes liveness and readiness probes
All existing WebSocket diagnostic commands (doctor.*, run.history, worker.list, etc.) continue to work

Implementation Notes

New instrumentation required

No throughput or latency metrics are tracked today. A sliding-window RunMetricsTracker needs to be added and wired into the run completion lifecycle — specifically the scheduled execution event handlers (schedule_completed/schedule_failed), webhook event handlers (webhook_completed/webhook_failed), and detached-run deregistration points.

Datastore health checking

No DatastoreVerifier is held in the serve handler's closure today — the doctor commands create verifiers on-the-fly. The health collector needs either a factory function or a persistent verifier instance constructed from datastoreConfig at startup.

Auth helper extraction

The bearer token + admin policy check pattern is currently inlined in the cancel endpoint. With three new endpoints using the same pattern, it should be extracted into a reusable helper to reduce duplication.

No new SSE library needed

Deno's native ReadableStream is sufficient for SSE — it's a text/event-stream response with formatted text pushed via a stream controller. No external dependencies required.

Alternatives Considered

WebSocket instead of SSE: The codebase already has mature WebSocket infrastructure (~70 commands). However, SSE is a better fit for this use case: it's read-only server-to-client push, requires no handshake negotiation, and has native support in curl, browser EventSource, and monitoring tools. Adding a WebSocket command would force dashboard integrators to implement the full WebSocket protocol.

Fleet aggregation: Each node could aggregate health from all peers via the control-plane store. This was rejected in favor of per-node reporting because it avoids leader-election complexity and lets the UI compose the fleet view with full control over layout and refresh cadence. The instanceId on each response provides a stable correlation key.

Full run history on /api/v1/health: Including all completed runs on the main health endpoint would bloat the response on busy nodes. The three-tier split (aggregate counters on health, full history on internal endpoint, existing run.history on WebSocket) gives operators the right level of detail at each layer.

02Bog Flow

Triaged

8/11/2026, 11:18:43 PM

Click a lifecycle step above to view its details.

03Sludge Pulse

stack72 assigned stack728/11/2026, 11:15:37 PM

Client shows 'Authentication failed' when actually rate-limited

Workers retry forever on permanent auth failures instead of backing off

Docs: add swamp access group list-idp to access commands reference

serve telemetry identity warnings are diagnostic but not actionable — say how to fix them

swamp quest: drop the heartbeat field the server no longer sends

--grant-reload auto does not detect filesystem changes — blind to ConfigMap updates

Server token definitions are not durable across pod restarts

WebSocket auth rate limiter blocks legitimate users when sharing IP with a failing client

Add native Bedrock Knowledge Base Retrieve support

GC expired OAuth server tokens and control plane secrets

migrate-index silently drops entries with shallow paths — all of workflows-evaluated/ and any root-level file

Cluster health snapshots with SSE streaming for swamp serve

Add swamp access group list-idp to enumerate IdP groups seen by the server

Grants directory loader skips symlinks — breaks Kubernetes ConfigMap mounts

Daemon telemetry flush: stop() can skip its final flush when a timer drain is already in flight

Support --grants-dir for loading grants from a directory

Docs: `swamp compile` subcommand doesn't exist

Guard the projector's full rewrites against the fail-open lease (ADV-6 from #1575/#1576)

Projector write amplification: board_totals/owner_devices full rewrites are owners × runs/day

Profile candidateIds prune costs a full ledger scan to build itself — 42.88M rows for one keyed read

Leaderboard phase 3: owner-grain board_totals so a cache miss is survivable at 10B

swamp serve is telemetry-dark: scheduled workflow runs never reach /ingest, and the daemon never flushes

CLI has never sent insert_id: retried telemetry batches are double-counted into cli_daily, inflating XP in an append-only ledger

Catalog backfill reconciles against the local cache, not the datastore

Extension bundle cache not invalidated when an imported sibling module changes (entry-file fingerprint only)

swamp serve: data.query fails permanently with "UNIQUE constraint failed: catalog.*" — one duplicate catalog 5-tuple aborts every catalog rebuild

dailyHistory range pushdown — deferred from #1576, and the measurement says why

score-rollup projector runs N× per cadence — the run gate is per-replica memory, not shared state

Docs: manual never mentions telemetry, and the serve/scheduling pages imply automated runs are visible when they are not

Explore: CLI binary update integrity lacks independent verification

Leaderboard phase 2: take score_grants off the board read path — read today from score_daily

Explore: reconcile quality rubric's dormant provenance factor

Explore: pull-time signature verification for extensions

Explore: transparency-log integration (Rekor) for extension signatures

Explore: publish-time artifact signing for extensions (cosign keyless)

Profile day drill-down returns 0 entries at the 5s ClickHouse ceiling for every operative

A ledger read failure still renders as 'no activity': loadProfileScore swallows to an empty standing behind HTTP 200

reconcile_from_disk_bench.ts fails deno check (stale DenoRuntime stub)

Improve multi-word search query handling and guidance

Retire the genesis_public flag and delete the dead cutover transcription scaffolding

Leaderboard phase 1: cache the board read and bust it on tier crossings (no schema change)

Catalog backfill deletes rows for any model whose data names are numeric

Docs: document catalog completeness check and --repair in doctor reference

Leaderboard board windows re-anchor on wall-clock while the rest of the view is frozen at asOf

Decision: move per-event grant MVs to day-grain — ledger is on track for ~1.5B score_grants / ~4B quest_grants

Leaderboard phase 5: score-rollup-projector write cost at 10B — ~430M rows scanned per 60s tick

Epic: /leaderboard and /u/{username} do not survive 10B events — measured audit + phased plan

Datastore-managed configuration for multi-instance serve deployments

Ghosts on the leaderboard 500

swamp command doesn't check cwd

swamp auth login process fails when run remotely

serve boot OOM: findAllGlobalSince loads all workflow runs as full aggregates during reconciliation

extension search: channel-filtered results carry null version fields, and multiple --channel values collapse instead of unioning

s3-datastore: localHasAllRemoteEntries uses bare rel instead of localRelPath, always returns false in namespaced mode

datastoreSetupExtension migrates .swamp/ to cache root instead of namespace directory

swamp serve writes outputs/ outside the bound namespace — silently never persisted to the remote datastore

s3-datastore: bulk pull writes to cache root instead of the namespace, wedging push and arming a full-datastore delete

git: add pull, fetch --tags, and cherry-pick methods

Workflow runs with wide dataArtifact fan-out (~400/run) OOM both workflow history get and swamp serve boot, crash-looping the scheduler; rollback doesn't help (state-triggered, not release-triggered)

Show user:<name> instead of user:<id> in grant list output

Reusable long-lived invite link: one URL to bring people into the swamp

Exclude swamp internal model types from model list output

Skill docs use invalid two-argument 'vault get' form

access token mint prints an invalid retrieval command (vault get, not vault read-secret)

serve reload and model edit timeout over --server in Kubernetes

swamp auth whoami should report paid tier and trial status

Use whoami entitlement to explain private-extension push refusals

Document --restricted-commands flag for swamp serve

vault put: pass tags through to extension provider CreateSecret/setSecret calls

Docs: update manual for 23 new --server commands

Update VaultProvider interface in @systeminit/swamp-testing to support optional tags in put()

Docs: document --grants-file flag for swamp serve

Docs: document --server support for serve reload

Docs: add --tag flag to vault put reference

Docs: update manual for --detach-runs deprecation and HA auto-detection

Document the Codex plugin in swamp docs

Add --server support to CLI commands missing it

Document the swamp Claude Code plugin in the swamp documentation

Make --server a global CLI option

Add --server support to model cancel (server-side run cancellation)