Skip to content

fix(benchmark): sglang cache-mix accounting fixes and unified cache-metrics sampler - #5

Merged
tigrawap merged 9 commits into
mainfrom
feat/sglang-endpoint-support
Nov 15, 2026
Merged

tigrawap merged 9 commits into
mainfrom
feat/sglang-endpoint-support

Conversation

@oferki

@oferki oferki commented Aug 28, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to the sglang endpoint support already on main: fixes cache-mix accounting bugs and unifies the vLLM/sglang cache-metrics sampler.

Fixes

  • Series born mid-run were dropped. fold() treated any first-sighted counter key as a baseline, so a tier whose label appears mid-run (sglang host/storage appear at their first nonzero hit) lost its accumulated value on an already-seen endpoint. The tier band was understated and compute overstated. A known endpoint's new key is now credited in full.
  • storage_* cache_source labels are folded into one storage series (otherwise their tokens landed in compute), and unknown cache_source values warn once.
  • Bare host:port metrics URLs (--vllm-metrics-url / --sglang-metrics-url) get an http:// scheme instead of failing at request time.
  • Mixed-engine traces are differenced per (model, engine) stream. Cumulative counters from different engines in one file were previously folded together, which produced wrong deltas when a run spanned vLLM and sglang.
  • 3D cache-mix categories cover every canonical band. gpu and external were missing from the 3D mapping. vLLM's local cache hits (reported as host) are drawn as GPU cache rather than DRAM, since vLLM's local cache is the GPU prefix cache; sglang host stays DRAM.
  • A partial scrape counts as a successful probe. When only some of several --*-metrics-url endpoints answer, the round is a success (so a spec that didn't say type=openai_vllm no longer gives up after 3 partial rounds), and the "N of M endpoints answered" log appears once per streak.

Refactor

  • One metricsSampler parameterized by a per-engine table replaces the separate vLLM and sglang samplers, with one report builder and one shared mixBandStyleJS.
  • Samples are written as cache_metrics_sample rows with canonical bands (compute, gpu/host/storage/external) and an engine field. Legacy vllm_metrics_sample / sglang_metrics_sample rows are translated on read, so old trace directories still render. docs/replay.md describes the new rows.
  • Report change: vLLM local_cache_hit is shown as host, and vLLM external has its own color, distinct from sglang storage.

Also

  • Adds tests for the --sglang-metrics-url override end to end and the no-total-label shape (test:), plus regression tests for each fix above.

Testing

  • task verify (gofmt, go vet, go test -race) passes on every commit in the series.
  • Regression testing: ran wekai benchmark with vllm and with sglang. Verified test runs succesfully and reports are coherent.

🤖 Generated with Claude Code


Note

Medium Risk
Large refactor of benchmark metrics I/O and report embedding affects trace compatibility (mitigated by legacy JSONL translation) and cache-mix accuracy for fleet restarts and mixed engines.

Overview
Unifies vLLM and SGLang cache-metrics sampling into a single metricsSampler in cache_metrics.go, with engine-specific parsing kept in vllm_metrics.go / sglang_metrics.go via cacheEngine tables. Benchmark runs now write cache_metrics_sample JSONL rows with canonical bands (compute, gpu, host, storage, external); reports use one buildSampleViz, shared mixBandStyleJS, and read-time translation of legacy vllm_metrics_sample / sglang_metrics_sample files.

Accounting and parsing fixes: delta fold() now credits counter keys that appear mid-run on an already-scraped endpoint (fixes understated sglang host/storage and overstated compute). SGLang parseSGLangCounters folds storage_* labels into storage and warns once on unknown cache_source values. normalizeMetricsURLs prepends http:// for bare host:port override URLs.

Report behavior: vLLM local cache maps to host; external gets a distinct color from sglang storage. Mixed-engine traces diff per (model, engine) stream; hostEngine drives 3D cache-mix categorization. readJSONLFileWithParams returns a single sample slice (signature change across callers/tests).

Reviewed by Cursor Bugbot for commit da9825e. Bugbot is set up for automated code reviews on this repo. Configure here.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread benchmark/auto.go Resolved

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread benchmark/replay.go Resolved

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread benchmark/auto.go Resolved
Comment thread benchmark/cache_metrics.go Resolved
oferki added a commit that referenced this pull request Oct 15, 2026
…to dataset-replay and synthetic paths

Two independently-flagged bugs (human review + Bugbot on PR #5):

1. cacheWarning latched permanently from one bad window. The periodic
   check used CacheMetrics' sliding window (last ~CacheWindowSize
   records): an all-cold-start window (common right after a series
   scale-up, or a pass boundary under --replay-reuse-sessions) is, by
   construction, a window with hitRate==0 and serverReported==false —
   cold starts are a structural miss, never a hit — and the old code
   latched st.cacheWarning=true the instant that happened, with nothing
   ever clearing it. A run could finish with a clearly non-zero
   DisplayHitRate right next to a "server may not support prompt
   caching" warning.

   Fixed by replacing the sticky per-window latch with
   noCacheEverObserved, computed from completionStream's existing
   all-time running counters (globalCachedTokens / globalInputWarm via
   TokenTotals) rather than a window. Those counters only grow, so the
   signal can only ever go true->false as real cache activity
   accumulates, never latch the other way. Applied uniformly: the live
   per-model status indicator, the final run summary, and the
   multi-model final snapshot all now derive cacheWarning from the same
   run-wide computation instead of a stored, latch-only bool. Removed
   the now-dead st.cacheWarning field and its periodic
   noCacheDataObserved(...)-triggered latch; noCacheDataObserved itself
   is untouched (it's a correct, separately-tested windowed primitive —
   TestNoCacheDataObserved still locks down its own semantics).

2. UsageObserved (drives the openai_sglang "no usage ever reported"
   warning) was only ever set by the router-replay path
   (consumeOpenAISSE/consumeOpenAIPlain), which can nil-check the raw
   wire JSON's "usage" key. Two other paths build RequestMetrics and
   never touched the field:
     - Dataset-replay (replay.go's flush()): populated UsageData from
       response.Usage but never set UsageObserved, so a dataset-replay
       run could never trip the warning no matter what the server did.
     - Synthetic recorder (bench_prompts.go's runSingleRequest): same
       gap, PLUS the synthetic recording block in auto.go never updated
       autoState.sawResponseEver/sawUsageEver at all (only
       recordReplayRequest did), so even with UsageObserved set
       correctly, a synthetic run's signal would always read
       "unobserved" as false regardless of the server's actual
       behavior.

   Both paths go through llm.Chat.Request, whose Response.Usage is a
   plain value (not a pointer) — always "present" at its zero value
   even when the server never reported usage — so there's no presence
   signal to nil-check the way the router-replay wire parsers can.
   Added usageCountsReported(prompt, completion, cached, reasoning) as
   the stand-in: "at least one usage count is nonzero" means the server
   actually reported something. Wired into both replay.go and
   bench_prompts.go. Also extracted the sawResponseEver/sawUsageEver
   update (previously inlined only in recordReplayRequest) into a
   shared trackUsageObserved(st, isErr, skipped, usageObserved) helper,
   now called from both recordReplayRequest and the synthetic path's
   recording block, so every run type feeds the same signal.

   Engine-gating: left isOpenAISGLangSpec's exclusive gate on the
   no-usage warning as-is, i.e. did NOT generalize it to fire for every
   engine. TestIsOpenAISGLangSpec's own doc comment explicitly asserts
   "it must fire only for type=openai_sglang, never for vLLM, plain
   openai, or a non-dynamic model spec" — this is a documented, tested
   design decision already in the codebase, not an oversight, so
   generalizing it would contradict an existing test contract rather
   than fix a bug. The part that WAS an oversight (dataset-replay and
   synthetic runs never feeding the signal at all) is fixed; which
   engines the warning applies to once the signal is correct is
   unchanged.

Regression tests added:
  - TestNoCacheEverObservedUnitCases / TestNoCacheEverObservedSurvivesColdStartWindow
    (throughput_test.go): an all-cold-start window followed by windows
    with real server-confirmed and heuristic cache hits; asserts the
    final signal is not latched true.
  - TestUsageCountsReported, TestDatasetReplayUsageObservedWiring,
    TestTrackUsageObservedSyntheticPath (replay_usage_observed_test.go):
    real usage data -> UsageObserved true / warning suppressed; usage
    genuinely absent -> UsageObserved false / warning still fires; same
    invariant for the synthetic path's trackUsageObserved call.

Verified: gofmt -l (clean), go vet ./... (clean), task verify (gofmt +
vet + full race suite, all packages ok).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread benchmark/cache_metrics.go Resolved
oferki added a commit that referenced this pull request Oct 15, 2026
…to dataset-replay and synthetic paths

Two independently-flagged bugs (human review + Bugbot on PR #5):

1. cacheWarning latched permanently from one bad window. The periodic
   check used CacheMetrics' sliding window (last ~CacheWindowSize
   records): an all-cold-start window (common right after a series
   scale-up, or a pass boundary under --replay-reuse-sessions) is, by
   construction, a window with hitRate==0 and serverReported==false —
   cold starts are a structural miss, never a hit — and the old code
   latched st.cacheWarning=true the instant that happened, with nothing
   ever clearing it. A run could finish with a clearly non-zero
   DisplayHitRate right next to a "server may not support prompt
   caching" warning.

   Fixed by replacing the sticky per-window latch with
   noCacheEverObserved, computed from completionStream's existing
   all-time running counters (globalCachedTokens / globalInputWarm via
   TokenTotals) rather than a window. Those counters only grow, so the
   signal can only ever go true->false as real cache activity
   accumulates, never latch the other way. Applied uniformly: the live
   per-model status indicator, the final run summary, and the
   multi-model final snapshot all now derive cacheWarning from the same
   run-wide computation instead of a stored, latch-only bool. Removed
   the now-dead st.cacheWarning field and its periodic
   noCacheDataObserved(...)-triggered latch; noCacheDataObserved itself
   is untouched (it's a correct, separately-tested windowed primitive —
   TestNoCacheDataObserved still locks down its own semantics).

2. UsageObserved (drives the openai_sglang "no usage ever reported"
   warning) was only ever set by the router-replay path
   (consumeOpenAISSE/consumeOpenAIPlain), which can nil-check the raw
   wire JSON's "usage" key. Two other paths build RequestMetrics and
   never touched the field:
     - Dataset-replay (replay.go's flush()): populated UsageData from
       response.Usage but never set UsageObserved, so a dataset-replay
       run could never trip the warning no matter what the server did.
     - Synthetic recorder (bench_prompts.go's runSingleRequest): same
       gap, PLUS the synthetic recording block in auto.go never updated
       autoState.sawResponseEver/sawUsageEver at all (only
       recordReplayRequest did), so even with UsageObserved set
       correctly, a synthetic run's signal would always read
       "unobserved" as false regardless of the server's actual
       behavior.

   Both paths go through llm.Chat.Request, whose Response.Usage is a
   plain value (not a pointer) — always "present" at its zero value
   even when the server never reported usage — so there's no presence
   signal to nil-check the way the router-replay wire parsers can.
   Added usageCountsReported(prompt, completion, cached, reasoning) as
   the stand-in: "at least one usage count is nonzero" means the server
   actually reported something. Wired into both replay.go and
   bench_prompts.go. Also extracted the sawResponseEver/sawUsageEver
   update (previously inlined only in recordReplayRequest) into a
   shared trackUsageObserved(st, isErr, skipped, usageObserved) helper,
   now called from both recordReplayRequest and the synthetic path's
   recording block, so every run type feeds the same signal.

   Engine-gating: left isOpenAISGLangSpec's exclusive gate on the
   no-usage warning as-is, i.e. did NOT generalize it to fire for every
   engine. TestIsOpenAISGLangSpec's own doc comment explicitly asserts
   "it must fire only for type=openai_sglang, never for vLLM, plain
   openai, or a non-dynamic model spec" — this is a documented, tested
   design decision already in the codebase, not an oversight, so
   generalizing it would contradict an existing test contract rather
   than fix a bug. The part that WAS an oversight (dataset-replay and
   synthetic runs never feeding the signal at all) is fixed; which
   engines the warning applies to once the signal is correct is
   unchanged.

Regression tests added:
  - TestNoCacheEverObservedUnitCases / TestNoCacheEverObservedSurvivesColdStartWindow
    (throughput_test.go): an all-cold-start window followed by windows
    with real server-confirmed and heuristic cache hits; asserts the
    final signal is not latched true.
  - TestUsageCountsReported, TestDatasetReplayUsageObservedWiring,
    TestTrackUsageObservedSyntheticPath (replay_usage_observed_test.go):
    real usage data -> UsageObserved true / warning suppressed; usage
    genuinely absent -> UsageObserved false / warning still fires; same
    invariant for the synthetic path's trackUsageObserved call.

Verified: gofmt -l (clean), go vet ./... (clean), task verify (gofmt +
vet + full race suite, all packages ok).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@oferki
oferki force-pushed the feat/sglang-endpoint-support branch from 9a02c87 to 2a5595d Compare October 15, 2026 21:01

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread benchmark/auto.go Resolved
@oferki
oferki force-pushed the feat/sglang-endpoint-support branch from eae68b1 to beea339 Compare November 4, 2026 19:16
@oferki oferki changed the title Feat/sglang endpoint support fix(benchmark): sglang cache-mix accounting fixes and unified cache-metrics sampler Nov 4, 2026
@oferki
oferki requested a balanced review from Copilot November 4, 2026 19:34

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Active issues misclassify GPU and external cache hits as recomputation in the visualization and inflate cache-mix totals with artificial spikes when SGLang and vLLM samples are differenced together.

2 open issues
What changed in this PR

This PR fixes cache-mix accounting and unifies vLLM and SGLang metrics sampling within the benchmark reporting pipeline, while preserving support for legacy traces.

Changes:

  • Corrects mid-run counter accounting, normalizes SGLang storage labels, and accepts bare metrics endpoints.
  • Introduces shared sampling, trace records, and report styling with legacy record translation.
  • Consolidates regression tests and removes an obsolete release script.
File Description
README.md Documents unified records and canonical cache bands.
create_router_release.sh Removes obsolete router release script.
benchmark/​vllm_metrics.go Defines vLLM parsing and canonical band mapping.
benchmark/​vllm_metrics_test.go Adapts vLLM tests to shared sampling.
benchmark/​vllm_metrics_delta_test.go Removes tests consolidated into shared coverage.
benchmark/​visualize.go Consumes unified samples and translates legacy records.
benchmark/​visualize_samples_test.go Updates fixtures and mixed-engine report checks.
benchmark/​visualize_public.go Uses shared cache-band styling.
benchmark/​visualize_public_test.go Updates public-report fixtures.
benchmark/​visualize_merge.go Merges unified metrics records.
benchmark/​timeline_test.go Adapts to the revised JSONL reader signature.
benchmark/​sglang_metrics.go Normalizes storage labels and defines SGLang bands.
benchmark/​sglang_metrics_test.go Covers label handling, derivation, and overrides.
benchmark/​run_params.go Updates the sample-type reference in a comment.
benchmark/​run_params_test.go Adapts reader calls to unified samples.
benchmark/​cache_metrics.go Adds shared sampling, accumulation, translation, and styling.
benchmark/​cache_metrics_test.go Consolidates sampler and compatibility regression coverage.
benchmark/​auto.go Updates a writer comment to name the unified sample type.

Review effort: Balanced

Comment thread benchmark/cache_metrics.go Outdated
Comment on lines +784 to +787
prev := sorted[i-1]
bands := make([]vizBand, 0, len(bandNames))
for _, name := range bandNames {
bands = append(bands, vizBand{Name: name, Tokens: clamp(smp.Bands[name], prev.Bands[name])})
Comment thread benchmark/visualize.go
Comment on lines +407 to +408
mix, adt = buildSampleViz(samples)
bandOrder = sampleBandNames(samples)

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit d226452. Configure here.

Comment thread benchmark/visualize.go Resolved
Comment thread benchmark/visualize.go Resolved
oferki and others added 9 commits November 12, 2026 07:14
…total-label shape

Adds tests only; the --sglang-metrics-url override and the compute
computation they exercise are already in the tree.

- TestStartSGLangMetricsSamplerUsesOverride: an override URL is polled
  instead of the model-derived endpoint, and the resulting sample is written
  to the JSONL stream.
- TestSGLangComputeWithoutTotalLabel: compute is derived correctly when the
  server exports no untiered `total` cache_source label.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ampler

fold() treated any first-sighted counter key as a baseline contributing
nothing, which is only correct for an endpoint's OWN first scrape. sglang's
host/storage cache_source labels are born mid-run at their first nonzero hit,
so the first scrape to see one on an already-seen endpoint discarded its
whole accumulated value. prompt_tokens_total had already counted those
tokens, so they landed in compute: sglang host/storage bands were understated
and compute overstated. fold() now distinguishes "endpoint's first scrape
ever" (a true baseline) from "known endpoint, new key" (credit the full
value). vLLM never hit this because it pre-creates every source label at
process start.

The bug was a symptom of vLLM and sglang each carrying a near-literal copy of
the sampler (struct, poll loop, Prometheus parser, delta accumulation,
on-disk record, report builder), with chart code branching per engine. The
fix is made once by collapsing them into a single cacheEngine
struct-of-functions table (vllmEngine/sglangEngine), one metricsSampler
parameterized by it, and one buildSampleViz/canonicalBandOrder pair that
drives the report off each sample's own Bands map.

Report and data changes:
- vLLM bands are relabeled onto the canonical set: local_compute -> compute,
  local_cache_hit -> host (vLLM does not split GPU- from CPU-resident local
  cache), external_kv_transfer -> external, labeled "external connector".
- vLLM external gets its own color, distinct from sglang storage (#7C03EC),
  since external_kv_transfer means any KV connector, not specifically WEKA
  L3; a merged vLLM+sglang report no longer renders the two identically.
- The duplicated MIX_BAND_STYLE/MIX_LEGEND_ORDER JS tables in visualize.go and
  visualize_public.go become one shared mixBandStyleJS string.
- Samples are written as cache_metrics_sample JSONL rows for both engines.
  Legacy vllm_metrics_sample / sglang_metrics_sample rows are translated on
  read, so old trace directories keep rendering.

The sglang storage_* label prefix fold and bare host:port metrics URLs land
in the next two commits.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…on unknown labels

sglang versions before upstream PR #37503 emit cache_source=f"storage_{backend}"
(e.g. "storage_HiCacheFile") instead of plain "storage". parseSGLangCounters now
folds any "storage"-prefixed label into the single "storage" key before the
delta-fold ever sees it, so a label-spelling change mid-fleet (rolling upgrade)
doesn't look like a brand-new series and break per-key continuity.

Also adds a one-time-per-process warning (not per-scrape) for any cache_source
value outside the known {device, host, storage*, total} vocabulary, so a future
unrecognized label doesn't silently vanish from every report band the way
storage_HiCacheFile did before this fix — its tokens are still parsed and
accumulated, just flagged as unmapped.
… a scheme

A bare "host:port" (no "://") previously produced a schemeless URL that failed
at request time with "unsupported protocol scheme", even though the
--vllm-metrics-url/--sglang-metrics-url CLI help text has always claimed a bare
endpoint is accepted. Now a bare host:port is assumed http:// before the rest
of the /metrics-suffix normalization runs.
The sampler section still named the per-engine record types and their
sources.* fields; samples are now cache_metrics_sample rows with bands.*.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
mixCat still switched on pre-unification band names, so the canonical
gpu and external bands fell through to compute and GPU radix hits and
KV-connector hits rendered as recomputation.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A generic spec probed for vLLM plus an explicit sglang URL writes both
engines' cumulative samples into one file. buildSampleViz diffed them as
a single series, so every engine switch read as a counter jump or reset.
Difference each (model, engine) stream against its own previous sample,
spread its delta over the intervals it spans, and sum the streams per
band.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The unified sampler relabels vLLM local_cache_hit as the host band, which
the 3D fill sent to DRAM. Each series now carries the engine whose host
tokens dominate (hostEngine) and mixCat maps host to hbm for vllm, dram
otherwise (sglang HiCache L2). The 2D labels and on-disk schema are
unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A round where only some endpoints answered bumped consecFails but never
reached noteSuccess, so a speculative sampler behind a partly reachable
fleet never retired its budget, never logged that it serves the counter
family, and was switched off by the first total outage after three partial
rounds. A partial answer proves the endpoints serve metrics: it now counts
as a success while EndpointsOK still records the coverage, and consecFails
only counts rounds where nothing answered.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@oferki
oferki force-pushed the feat/sglang-endpoint-support branch from 7e98b8e to da9825e Compare November 12, 2026 18:39
@tigrawap
tigrawap merged commit 7f3cbea into main Nov 15, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants