Skip to main content
← Back to list
01Issue
FeatureOpenSwamp Club
AssigneesNone

Relationships

#1578 Decision: move per-event grant MVs to day-grain — ledger is on track for ~1.5B score_grants / ~4B quest_grants

Opened by keeb · 8/10/2026

Phase 6 of #1572. This is a product decision, not an engineering task — but it gates how hard #1577 (the projector write side) has to be, so it should be decided early even if implemented late.

The observation

score_grants grows roughly 1:1 with events, because several MVs mint one ledger row per event with no aggregation:

  • score_grants_from_pull_mv — grant_id = e.insert_id, no GROUP BY
  • quest_grants_drip_model_mv — grant_id = e.insert_id, no GROUP BY
  • quest_grants_drip_workflow_mv — same
  • quest_grants_drip_pull_mv — same
  • quest_grants_spike_publish_mv — same

Measured ratio on the dev dataset: 0.145 grants per event (17.35M events -> 2.52M score_grants), plus 6.82M quest_grants.

Projected to 10B events: ~1.5B score_grants rows and ~4B quest_grants rows. For calibration on how badly this already skews, quest_grants currently has a single identity holding 18.6M rows.

That substrate size is the input to every read in this epic and to the projector in #1577.

The precedent already in the codebase

score_grants_from_presence_mv already does exactly the right thing:

GROUP BY distinct_id, day

One row per device-day instead of one per event. The pattern is proven, in production, on the same insert path.

The proposal to decide on

Move the per-event grant MVs to day-grain, following the presence MV's shape. Estimated substrate reduction: ~30x.

Consequences worth weighing explicitly:

  • extension_pull grants lose nothing narratively. They are already excluded from the combat log by design (a per-pull row narrates who pulls which extension in near-real-time, which leaks registry usage). So day-grain costs no visible detail there.
  • Quest drip grants are the harder call. They are the mechanism behind quest progress, and collapsing them to day-grain changes when and how progress accrues. This is where the product judgement actually lives.
  • Score values change. Not necessarily in total, but potentially in timing and in how the combat log reads. That is user-visible and perceptually irreversible, which is why this is filed as a decision rather than a task.
  • Existing history. A grain change applies going forward; whether to re-grain history, and what that does to standings, is part of the decision.

What this is not

This is not a proposal to move the ledger out of ClickHouse, and not a proposal to freeze identity into grants. The ledger stays where it is, keyed by distinct_id, with balance as a read-time aggregate over a mutable identity set. Only the grain of the row is in question.

It is also not an anti-farming measure and should not be argued as one. The scoring stance is settled; this is purely about substrate size.

Why decide it early

#1577 (projector write side) is the largest and riskiest phase in the epic, and its difficulty is a direct function of ledger size. If the ledger shrinks ~30x, much of #1577 becomes tuning. If it does not, #1577 is a re-architecture. Committing to #1577's shape before this is decided risks building the wrong thing.

Suggested next step

Not implementation — a written decision, with the quest-drip consequence spelled out concretely enough that the scoring implications are visible before any MV is touched.

02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

8/10/2026, 7:20:48 PM

No activity in this phase yet.

03Sludge Pulse

Sign in to post a ripple.