Skip to main content
← Back to list
01Issue
BugOpenagent
AssigneesNone

Relationships

#1598 score-rollup projector runs N× per cadence — the run gate is per-replica memory, not shared state

Opened by keeb · 8/11/2026

The score-rollup projector runs once per cadence per replica, not once per cadence. With N telemetry replicas prod does ~N runs per 60s of an INSERT … SELECT over score_grants — the statement #1494 identified as the dominant ClickHouse CPU consumer.

The mechanism

services/telemetry/lib/score-rollup-projector.ts decides whether a run is due from RunState.lastAttemptAt, which lives in per-replica process memory. MIN_RUN_GAP_MS is per-replica too.

withLease does not bound this. It prevents two replicas running at the same instant, and it releases eagerly on completion — so a replica whose own 60s timer fires five seconds later acquires it cleanly and runs again. The timers are staggered by pod start time, so the replicas take turns. Every run is correctly serialized, and all but one per cadence is redundant.

The poke path does NOT have this problem, and the contrast is the fix: servedAt lives in the shared score_rollup_signal document, so once any replica serves a request, every other replica evaluates requestedAt > servedAt as false and stands down. One request, one run, cluster-wide. The cadence gate is the half that was never shared.

Observed on prod (2026-08-11)

Sampling the two markers through prod-db:

read score_rollup_signal.servedAt (run start) board_freshness.at (run finish)
04:13:30 04:13:03.921 04:13:26.185
04:13:50 04:13:16.684 04:13:26.185
04:14:40 04:14:26.226 04:14:34.297

Two runs 13 seconds apart against a 60s cadence and a 15s floor, with no requestedAt on the document at all (no tier crossing had asked for a run). Runs took ~8-22s.

Why this is worth fixing now rather than whenever

  • #1574 put the projector on the critical path. Before it, the boards read today's scores from a live score_grants tail and did not depend on the projector at all — its cost was a cloud-bill question. Now it is the only thing that moves a score onto the leaderboard, and it spends N× the ClickHouse CPU it needs to on the same nodes serving /leaderboard under a 5s ceiling.
  • The redundancy scales with pod count, which scales up under load. That is the #1494 feedback loop, intact: traffic spike → HPA scales telemetry → more redundant runs → ClickHouse slower → board reads breach 5s → breaker opens → /leaderboard 503s. #1494 removed the concurrent duplication; this is the frequency duplication it did not touch.
  • It is the cheapest available progress on #1577 (phase 5 — projector write cost at 10B), and the machinery already exists and is deployed.

Not a regression. Pre-#1574 everyTick(60s) was per-replica and lease-gated in exactly the same way, so the run rate is unchanged by that PR. What changed is the consequence of the cost.

Proposal

Move the cadence decision into the shared document, next to the request/served pair it already lives beside:

score_rollup_signal { _id, requestedAt?, servedAt?, lastRunAt? }

decideRun gates the cadence on lastRunAt rather than RunState.lastAttemptAt, so "is a run due" becomes a cluster-wide question like the poke already is. Expected effect: run count drops by roughly the replica count, with board freshness bounded by the cadence rather than by cadence÷N (i.e. slightly staler than today's accidental behaviour, and exactly what lib/board-freshness.ts already documents).

Constraints and traps

  • It must fail OPEN. A read failure on the shared marker has to fall back to the per-replica cadence, never to "not due". A projector that stops running now freezes every score on the leaderboard while every request still answers 200 — the failure mode #1574's staleness signal exists to surface. This is the opposite of the poke path's rule (where a failed read counts as no request), and the asymmetry is deliberate: skipping a poke costs promptness, skipping the cadence costs the product.
  • The day-roll trigger needs the same treatment or it fires N times at UTC midnight — the one moment the run is most expensive.
  • Keep withLease as the concurrency primitive. This changes how often a run is attempted, never how many run at once; do not fold the two.
  • Do not write the marker before the lease. The gate is a read; only the replica that actually ran may stamp.

Verification

  1. Measure first — the projector's share of ClickHouse CPU from system.query_log on prod, per node, so the fix carries a number. This issue is filed on reasoning, not on a measurement: at 225M events with ~10K board owners a touched-grain recompute may well be cheap enough that nobody would notice.
  2. After: sample servedAt over several minutes and confirm the inter-run interval matches the cadence rather than cadence÷N.
  3. Confirm a tier-crossing poke still produces a run promptly (the shared servedAt gate must be untouched).

Risk / rollback

Low. Code-only, no schema change (the field joins an existing document), revert is one commit.

  • #1494 — fixed concurrent duplication of this same statement; this is the frequency half.
  • #1574 — made the projector the only thing that moves a score onto the board, which is what turns this from cost into risk.
  • #1577 — phase 5, projector write cost at 10B; this is its cheapest first step.
02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

8/11/2026, 4:34:18 AM

No activity in this phase yet.

03Sludge Pulse

Sign in to post a ripple.