Skip to content

bench(zero-cache): add rm vs load benchmark harness#6070

Draft
Karavil wants to merge 1 commit into
rocicorp:mainfrom
Karavil:capy/rm-vs-load-benchmark
Draft

bench(zero-cache): add rm vs load benchmark harness#6070
Karavil wants to merge 1 commit into
rocicorp:mainfrom
Karavil:capy/rm-vs-load-benchmark

Conversation

@Karavil

@Karavil Karavil commented May 26, 2026

Copy link
Copy Markdown
Contributor

BLUF: this PR gives the rest of the stack a benchmark that matches the deployment shape we actually care about. Its first output is a reproducible baseline for the old serving apply path: 247.0 tx/s, 4,939.9 rows/s, 2.561 ms avg VS transaction apply, and 0 sync worker read errors with eight readers sharing the serving SQLite file.

The production shape to keep in mind is:

RM machine                                  VS machine
upstream changes
  -> storer/changeLog
  -> websocket v6 stream  ----------------> serving replicator
                                               |
                                               v
                                          serving SQLite file
                                               ^
                                               |
                                      8 sync workers read here

The benchmark harness in #6070 models this as shared-replica-sync-workers: one VS-side stream consumer, one serving replica applier, eight reader workers on the same SQLite file, v6 websocket frames, per-message ACKs, and reconnect catchup during load. The point is to avoid measuring an easier but less useful shape where each sync worker gets treated like an independent SQLite writer.

This benchmark is worth adding because the real bottleneck can move between the RM and the VS. A fast RM does not help if the VS serving replica cannot apply and publish fresh versions. A fast VS does not help if the RM storer or fanout path cannot keep the stream moving. The harness keeps those parts in one run so we can see which change moved the actual RM to VS path.

The code is intentionally contained under bench/rm-vs-load. packages/zero-cache/bench/rm-vs-load/scenarios.ts defines the scenario, packages/zero-cache/bench/rm-vs-load/runner.ts wires together the RM, storer, forwarder, stream consumer, reconnect consumer, and readers, and packages/zero-cache/bench/rm-vs-load/sync-workers.ts starts the shared SQLite reader workers.

{
  name: 'shared-replica-sync-workers',
  rowsPerTx: 20,
  syncWorkerCount: env.int('ZERO_RM_VS_SYNC_WORKERS', 8),
  workload: {kind: 'insert-only'},
}

Baseline output:

metric old worker-message path
load tx/s 247.0
load rows/s 4,939.9
avg VS tx apply 2.561 ms
storer drain 4,193.2 ms
reconnect catchup to final 11,761.0 ms
sync worker reads/s 258.4
sync worker avg read 29.597 ms
sync worker max read 196.267 ms
sync worker errors 0

Validation:

pnpm exec vitest --project='*no-pg*' run bench/rm-vs-load/config.test.ts bench/rm-vs-load/scenarios.test.ts bench/rm-vs-load/workloads.test.ts
pnpm --filter zero-cache run lint
pnpm --filter @rocicorp/zero-events run build
pnpm --filter zero-cache run check-types

Stack context: #6070 adds the benchmark, #6071 reduces RM storer cost, #6072 reduces VS apply cost, #6073 reduces thread-worker handoff cost, #6074 removes the serving-mode thread hop, #6075 improves RM fanout flow control, and #6076 removes subscription queue shifting.

@vercel

vercel Bot commented May 26, 2026

Copy link
Copy Markdown

@Karavil is attempting to deploy a commit to the Rocicorp Team on Vercel.

A member of the Team first needs to authorize it.

Why: make RM to serving-replica perf work measurable and reviewable before changing production paths

* add a v6 RM to serving replica benchmark with reconnect catchup

* add config, scenario, and workload tests

* expose package scripts for smoke and e2e runs
@Karavil Karavil force-pushed the capy/rm-vs-load-benchmark branch from c3d3d0b to 6d98d20 Compare May 26, 2026 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant