Skip to main content
← Back to list
01Issue
BugTriagedSwamp CLI
Assigneesstack72

Relationships

#1604 Daemon telemetry flush: stop() can skip its final flush when a timer drain is already in flight

Opened by keeb · 8/11/2026

Summary

DaemonTelemetryFlushService.stop() performs a final flush so a draining swamp serve does not strand the runs it just finished. That final flush can silently do nothing: if a timer tick's #drain() is already running when stop() is called, the #running guard makes stop()'s drain an immediate no-op, stop() resolves, and shutdown proceeds.

Found by the adversarial review on swamp-club/swamp#2117 (shipped in v20260811.162323.0-sha.e1a01920). Not a regression — it arrived with the flush loop in that PR.

Detail

src/serve/telemetry_flush.ts:

async #drain(): Promise<void> {
  if (this.#running) return;   // <- makes stop()'s drain a no-op
  ...
}

async stop(): Promise<void> {
  // clear timer, set #stopped
  await this.#drain();          // <- may return immediately
}

Ticks are started as void this.#tick(), so nothing holds a reference to the in-flight drain and stop() cannot await it.

Concrete scenario

  1. T=0 — timer fires, #drain() starts and calls findUnflushed.
  2. T=0.5s — a workflow run completes and writes its telemetry entries.
  3. T=1s — shutdown begins; stop() is called. #running is still true, so its drain returns immediately.
  4. stop() resolves; serve.ts proceeds to ac.abort() and the CLI teardown.

The entries from step 2 were never in the batch the in-flight drain read at T=0.

Why it is not worse than it sounds

  • The in-flight drain loops until the spool is empty, so it often picks the new entries up on its next batch.
  • The CLI teardown flush is a second chance, though its AbortSignal.timeout(2000) starts at creation and can expire while serialized behind the in-flight drain on the shared #flushChain.
  • insert_id (added in the same PR) makes it harmless if both paths send the same entry.
  • serve.ts drains active runs before calling stop(), so the window is narrow.

Worst case is not data loss — unflushed entries stay on disk and go out on the next swamp invocation. But a daemon that is stopped and not restarted holds them indefinitely, which is the shape of the problem #1591 existed to fix.

Suggested fix

Track the in-flight drain so stop() can await it before its own:

#drainPromise: Promise<void> | null = null;

async stop(): Promise<void> {
  // ...clear timer, set #stopped...
  if (this.#drainPromise) await this.#drainPromise;
  await this.#drain();
}

A test would drive a tick, call stop() while that drain is still resolving, and assert that entries written in between are sent.

Also in the same area

src/domain/telemetry/telemetry_service.ts — the forkForRun docstring says flushing is serialized by a per-instance mutex {@link #flushInFlight}, but the field is #flushChain. Stale reference left by a rename during development, sitting in the comment that documents the concurrency invariant. Worth fixing alongside.

Environment

  • swamp v20260811.162323.0-sha.e1a01920 and later
  • Only affects swamp serve; the interactive CLI does not run the flush loop
  • #1591 — the serve telemetry gap this loop was built for
  • #1603 — insert_id idempotency, which bounds the blast radius here
02Bog Flow
✓OPEN◉TRIAGED○IN PROGRESS○SHIPPED+ 1 MOREASSIGNED+ 2 MOREREVIEW

Triaged

8/11/2026, 5:55:55 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack728/11/2026, 5:54:54 PM

Sign in to post a ripple.