Skip to main content
← Back to list
01Issue
FeatureTriagedSwamp CLI
Assigneeskeeb

Relationships

#1607 Guard the projector's full rewrites against the fail-open lease (ADV-6 from #1575/#1576)

Opened by keeb · 8/11/2026

Shipped knowingly with #1575/#1576 (PR #1061). The guard was in the approved plan and was not implemented.

What

withLease (services/telemetry/lib/leader-lease.ts) fails open by design: any Mongo error other than a lost CAS logs "lease acquisition failed, running unguarded" and runs the function anyway. That is deliberate and correct — a lease that cannot be acquired must never freeze score_daily finalization.

Before #1575/#1576 the cost of failing open was every telemetry replica running one idempotent INSERT … SELECT over the touched grain window. Now it is every replica running three statements, two of which are full rewrites of every owner.

So during a Mongo outage — the same outage that causes the fail-open in the first place — N replicas each rewrite board_totals and owner_devices in full, concurrently, and the HPA may well have scaled N up. That is #1494's shape with a bigger write, arriving at the worst possible moment.

What is NOT wrong

Correctness is unaffected. Every statement is idempotent and version-stamped, so concurrent runs converge on the same rows. This is purely cluster pressure.

Also note the structural mitigation that already exists and should not be double-counted: the requested run path cannot fire during a Mongo outage at all, because the signal read is the same Mongo and a failed read counts as no request. So the frequency during an outage falls back to the ~60s cadence, not the 10s poll. MIN_RUN_GAP_MS is the second belt.

What is missing is a guard on the work, not the frequency.

Proposed fix

A guard that does not depend on Mongo, because Mongo is the thing that is down. The cheapest form: skip the rewrite when the projection is already current.

  • Compare max(version) in board_totals against max(version) in score_daily (or against the run floor) and no-op when the projection is newer than the last rollup write.
  • Entirely ClickHouse-side, so it holds during exactly the outage that triggers the fail-open.
  • Bonus: it also addresses the sibling write-amplification issue in the steady state, since a tick where the rollup wrote nothing then costs one cheap comparison instead of two full rewrites.

That overlap is the reason to build the two together rather than separately.

Where

services/telemetry/lib/score-rollup-projector.ts, in runScoreRollupProjectorOnce — the loop over PROJECTION_SQL_URLS added by PR #1061. The per-statement span attributes (rollup.board_totals_ms, rollup.owner_devices_ms) are already in place, so the effect is measurable once the guard lands.

Parent: #1572. Sibling: the write-amplification issue. Recorded as ADV-6 in the #1575/#1576 plan review; step 6's risk line called for exactly this guard and it did not get built.

02Bog Flow
✓OPEN◉TRIAGED○IN PROGRESS○SHIPPED+ 1 MOREASSIGNED+ 3 MOREREVIEW

Triaged

8/11/2026, 5:39:46 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
keeb assigned keeb8/11/2026, 5:38:07 PM

Sign in to post a ripple.