Skip to content

perf(metrics): one pass per histogram, and scrapes share the running read - #77

Merged
gedaiu merged 1 commit into
mainfrom
metrics-scrape-cost
Nov 14, 2026
Merged

gedaiu merged 1 commit into
mainfrom
metrics-scrape-cost

Conversation

@gedaiu

@gedaiu gedaiu commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

Why

Every page that reads the floor took 18-21 s. pg_stat_activity on the shared lore-db showed fifteen copies of the claim-latency histogram running at once, the oldest for 93 s: /metrics joins every event (and every run and visit) to every bucket bound on each scrape, Prometheus scrapes each of the two api replicas every 15 s, so the scrapes stacked and starved the floor's own reads.

What Changed

  • histogram (packages/store/src/metrics-sql.ts): one pass, each row's bucket from width_bucket over the negated bounds (keeping seconds <= le), cumulated in foldHistogramRows.
  • sharedRead (apps/api/src/routes/shared-read.ts): a scrape that arrives while one is being answered waits for that answer; a failed read is not reused.

Testing

New store tests run the histogram SQL on the test database (inclusive bounds, label grouping); fold and shared-read unit tests; api suite 409 and store metrics tests pass; lint, typecheck, format clean.

🤖 Generated with Claude Code

… already running

Each /metrics scrape joined every event, run and visit ever recorded to
every bucket bound, and took over 90 s on the shared Postgres; with a
scrape every 15 s on two replicas, six piled up and slowed every floor
read. A histogram now buckets each row once with width_bucket, and a
scrape that arrives while one is answered shares its answer.
@lore-agent

lore-agent Bot commented Nov 14, 2026

Copy link
Copy Markdown

Lore is reviewing this PR — 29093c7d-9345-4bcb-9fe9-f7add5cb5f20.

Submit a review that requests changes to have Lore push a fix for it. Comment @lore review to re-run the review.

@lore-agent lore-agent Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lore review — Approved

Computes histogram buckets in a single pass via width_bucket over negated reversed bounds and coalesces concurrent /metrics scrapes via sharedRead while preserving Prometheus le bucket semantics.

Must fix (0) · Consider (0) · Nits (0)

Reviewed by gemini-4-argon

Submit a review that requests changes to have Lore push a fix for it. Comment @lore review to re-run the review.

Posted by floor, visit d7dc726f-36c7-4a4b-aa19-b06992f6a3d9.

@gedaiu
gedaiu merged commit 62cffb6 into main Nov 14, 2026
31 checks passed
@gedaiu gedaiu mentioned this pull request Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant