Repository navigation
fix(benchmark): sglang cache-mix accounting fixes and unified cache-metrics sampler - #5
Conversation
…to dataset-replay and synthetic paths Two independently-flagged bugs (human review + Bugbot on PR #5): 1. cacheWarning latched permanently from one bad window. The periodic check used CacheMetrics' sliding window (last ~CacheWindowSize records): an all-cold-start window (common right after a series scale-up, or a pass boundary under --replay-reuse-sessions) is, by construction, a window with hitRate==0 and serverReported==false — cold starts are a structural miss, never a hit — and the old code latched st.cacheWarning=true the instant that happened, with nothing ever clearing it. A run could finish with a clearly non-zero DisplayHitRate right next to a "server may not support prompt caching" warning. Fixed by replacing the sticky per-window latch with noCacheEverObserved, computed from completionStream's existing all-time running counters (globalCachedTokens / globalInputWarm via TokenTotals) rather than a window. Those counters only grow, so the signal can only ever go true->false as real cache activity accumulates, never latch the other way. Applied uniformly: the live per-model status indicator, the final run summary, and the multi-model final snapshot all now derive cacheWarning from the same run-wide computation instead of a stored, latch-only bool. Removed the now-dead st.cacheWarning field and its periodic noCacheDataObserved(...)-triggered latch; noCacheDataObserved itself is untouched (it's a correct, separately-tested windowed primitive — TestNoCacheDataObserved still locks down its own semantics). 2. UsageObserved (drives the openai_sglang "no usage ever reported" warning) was only ever set by the router-replay path (consumeOpenAISSE/consumeOpenAIPlain), which can nil-check the raw wire JSON's "usage" key. Two other paths build RequestMetrics and never touched the field: - Dataset-replay (replay.go's flush()): populated UsageData from response.Usage but never set UsageObserved, so a dataset-replay run could never trip the warning no matter what the server did. - Synthetic recorder (bench_prompts.go's runSingleRequest): same gap, PLUS the synthetic recording block in auto.go never updated autoState.sawResponseEver/sawUsageEver at all (only recordReplayRequest did), so even with UsageObserved set correctly, a synthetic run's signal would always read "unobserved" as false regardless of the server's actual behavior. Both paths go through llm.Chat.Request, whose Response.Usage is a plain value (not a pointer) — always "present" at its zero value even when the server never reported usage — so there's no presence signal to nil-check the way the router-replay wire parsers can. Added usageCountsReported(prompt, completion, cached, reasoning) as the stand-in: "at least one usage count is nonzero" means the server actually reported something. Wired into both replay.go and bench_prompts.go. Also extracted the sawResponseEver/sawUsageEver update (previously inlined only in recordReplayRequest) into a shared trackUsageObserved(st, isErr, skipped, usageObserved) helper, now called from both recordReplayRequest and the synthetic path's recording block, so every run type feeds the same signal. Engine-gating: left isOpenAISGLangSpec's exclusive gate on the no-usage warning as-is, i.e. did NOT generalize it to fire for every engine. TestIsOpenAISGLangSpec's own doc comment explicitly asserts "it must fire only for type=openai_sglang, never for vLLM, plain openai, or a non-dynamic model spec" — this is a documented, tested design decision already in the codebase, not an oversight, so generalizing it would contradict an existing test contract rather than fix a bug. The part that WAS an oversight (dataset-replay and synthetic runs never feeding the signal at all) is fixed; which engines the warning applies to once the signal is correct is unchanged. Regression tests added: - TestNoCacheEverObservedUnitCases / TestNoCacheEverObservedSurvivesColdStartWindow (throughput_test.go): an all-cold-start window followed by windows with real server-confirmed and heuristic cache hits; asserts the final signal is not latched true. - TestUsageCountsReported, TestDatasetReplayUsageObservedWiring, TestTrackUsageObservedSyntheticPath (replay_usage_observed_test.go): real usage data -> UsageObserved true / warning suppressed; usage genuinely absent -> UsageObserved false / warning still fires; same invariant for the synthetic path's trackUsageObserved call. Verified: gofmt -l (clean), go vet ./... (clean), task verify (gofmt + vet + full race suite, all packages ok). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…to dataset-replay and synthetic paths Two independently-flagged bugs (human review + Bugbot on PR #5): 1. cacheWarning latched permanently from one bad window. The periodic check used CacheMetrics' sliding window (last ~CacheWindowSize records): an all-cold-start window (common right after a series scale-up, or a pass boundary under --replay-reuse-sessions) is, by construction, a window with hitRate==0 and serverReported==false — cold starts are a structural miss, never a hit — and the old code latched st.cacheWarning=true the instant that happened, with nothing ever clearing it. A run could finish with a clearly non-zero DisplayHitRate right next to a "server may not support prompt caching" warning. Fixed by replacing the sticky per-window latch with noCacheEverObserved, computed from completionStream's existing all-time running counters (globalCachedTokens / globalInputWarm via TokenTotals) rather than a window. Those counters only grow, so the signal can only ever go true->false as real cache activity accumulates, never latch the other way. Applied uniformly: the live per-model status indicator, the final run summary, and the multi-model final snapshot all now derive cacheWarning from the same run-wide computation instead of a stored, latch-only bool. Removed the now-dead st.cacheWarning field and its periodic noCacheDataObserved(...)-triggered latch; noCacheDataObserved itself is untouched (it's a correct, separately-tested windowed primitive — TestNoCacheDataObserved still locks down its own semantics). 2. UsageObserved (drives the openai_sglang "no usage ever reported" warning) was only ever set by the router-replay path (consumeOpenAISSE/consumeOpenAIPlain), which can nil-check the raw wire JSON's "usage" key. Two other paths build RequestMetrics and never touched the field: - Dataset-replay (replay.go's flush()): populated UsageData from response.Usage but never set UsageObserved, so a dataset-replay run could never trip the warning no matter what the server did. - Synthetic recorder (bench_prompts.go's runSingleRequest): same gap, PLUS the synthetic recording block in auto.go never updated autoState.sawResponseEver/sawUsageEver at all (only recordReplayRequest did), so even with UsageObserved set correctly, a synthetic run's signal would always read "unobserved" as false regardless of the server's actual behavior. Both paths go through llm.Chat.Request, whose Response.Usage is a plain value (not a pointer) — always "present" at its zero value even when the server never reported usage — so there's no presence signal to nil-check the way the router-replay wire parsers can. Added usageCountsReported(prompt, completion, cached, reasoning) as the stand-in: "at least one usage count is nonzero" means the server actually reported something. Wired into both replay.go and bench_prompts.go. Also extracted the sawResponseEver/sawUsageEver update (previously inlined only in recordReplayRequest) into a shared trackUsageObserved(st, isErr, skipped, usageObserved) helper, now called from both recordReplayRequest and the synthetic path's recording block, so every run type feeds the same signal. Engine-gating: left isOpenAISGLangSpec's exclusive gate on the no-usage warning as-is, i.e. did NOT generalize it to fire for every engine. TestIsOpenAISGLangSpec's own doc comment explicitly asserts "it must fire only for type=openai_sglang, never for vLLM, plain openai, or a non-dynamic model spec" — this is a documented, tested design decision already in the codebase, not an oversight, so generalizing it would contradict an existing test contract rather than fix a bug. The part that WAS an oversight (dataset-replay and synthetic runs never feeding the signal at all) is fixed; which engines the warning applies to once the signal is correct is unchanged. Regression tests added: - TestNoCacheEverObservedUnitCases / TestNoCacheEverObservedSurvivesColdStartWindow (throughput_test.go): an all-cold-start window followed by windows with real server-confirmed and heuristic cache hits; asserts the final signal is not latched true. - TestUsageCountsReported, TestDatasetReplayUsageObservedWiring, TestTrackUsageObservedSyntheticPath (replay_usage_observed_test.go): real usage data -> UsageObserved true / warning suppressed; usage genuinely absent -> UsageObserved false / warning still fires; same invariant for the synthetic path's trackUsageObserved call. Verified: gofmt -l (clean), go vet ./... (clean), task verify (gofmt + vet + full race suite, all packages ok). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
9a02c87 to
2a5595d
Compare
eae68b1 to
beea339
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Active issues misclassify GPU and external cache hits as recomputation in the visualization and inflate cache-mix totals with artificial spikes when SGLang and vLLM samples are differenced together.
2 open issues
What changed in this PR
This PR fixes cache-mix accounting and unifies vLLM and SGLang metrics sampling within the benchmark reporting pipeline, while preserving support for legacy traces.
Changes:
- Corrects mid-run counter accounting, normalizes SGLang storage labels, and accepts bare metrics endpoints.
- Introduces shared sampling, trace records, and report styling with legacy record translation.
- Consolidates regression tests and removes an obsolete release script.
| File | Description |
|---|---|
| README.md | Documents unified records and canonical cache bands. |
| create_router_release.sh | Removes obsolete router release script. |
| benchmark/vllm_metrics.go | Defines vLLM parsing and canonical band mapping. |
| benchmark/vllm_metrics_test.go | Adapts vLLM tests to shared sampling. |
| benchmark/vllm_metrics_delta_test.go | Removes tests consolidated into shared coverage. |
| benchmark/visualize.go | Consumes unified samples and translates legacy records. |
| benchmark/visualize_samples_test.go | Updates fixtures and mixed-engine report checks. |
| benchmark/visualize_public.go | Uses shared cache-band styling. |
| benchmark/visualize_public_test.go | Updates public-report fixtures. |
| benchmark/visualize_merge.go | Merges unified metrics records. |
| benchmark/timeline_test.go | Adapts to the revised JSONL reader signature. |
| benchmark/sglang_metrics.go | Normalizes storage labels and defines SGLang bands. |
| benchmark/sglang_metrics_test.go | Covers label handling, derivation, and overrides. |
| benchmark/run_params.go | Updates the sample-type reference in a comment. |
| benchmark/run_params_test.go | Adapts reader calls to unified samples. |
| benchmark/cache_metrics.go | Adds shared sampling, accumulation, translation, and styling. |
| benchmark/cache_metrics_test.go | Consolidates sampler and compatibility regression coverage. |
| benchmark/auto.go | Updates a writer comment to name the unified sample type. |
Review effort: Balanced
| prev := sorted[i-1] | ||
| bands := make([]vizBand, 0, len(bandNames)) | ||
| for _, name := range bandNames { | ||
| bands = append(bands, vizBand{Name: name, Tokens: clamp(smp.Bands[name], prev.Bands[name])}) |
| mix, adt = buildSampleViz(samples) | ||
| bandOrder = sampleBandNames(samples) |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit d226452. Configure here.
…total-label shape Adds tests only; the --sglang-metrics-url override and the compute computation they exercise are already in the tree. - TestStartSGLangMetricsSamplerUsesOverride: an override URL is polled instead of the model-derived endpoint, and the resulting sample is written to the JSONL stream. - TestSGLangComputeWithoutTotalLabel: compute is derived correctly when the server exports no untiered `total` cache_source label. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ampler fold() treated any first-sighted counter key as a baseline contributing nothing, which is only correct for an endpoint's OWN first scrape. sglang's host/storage cache_source labels are born mid-run at their first nonzero hit, so the first scrape to see one on an already-seen endpoint discarded its whole accumulated value. prompt_tokens_total had already counted those tokens, so they landed in compute: sglang host/storage bands were understated and compute overstated. fold() now distinguishes "endpoint's first scrape ever" (a true baseline) from "known endpoint, new key" (credit the full value). vLLM never hit this because it pre-creates every source label at process start. The bug was a symptom of vLLM and sglang each carrying a near-literal copy of the sampler (struct, poll loop, Prometheus parser, delta accumulation, on-disk record, report builder), with chart code branching per engine. The fix is made once by collapsing them into a single cacheEngine struct-of-functions table (vllmEngine/sglangEngine), one metricsSampler parameterized by it, and one buildSampleViz/canonicalBandOrder pair that drives the report off each sample's own Bands map. Report and data changes: - vLLM bands are relabeled onto the canonical set: local_compute -> compute, local_cache_hit -> host (vLLM does not split GPU- from CPU-resident local cache), external_kv_transfer -> external, labeled "external connector". - vLLM external gets its own color, distinct from sglang storage (#7C03EC), since external_kv_transfer means any KV connector, not specifically WEKA L3; a merged vLLM+sglang report no longer renders the two identically. - The duplicated MIX_BAND_STYLE/MIX_LEGEND_ORDER JS tables in visualize.go and visualize_public.go become one shared mixBandStyleJS string. - Samples are written as cache_metrics_sample JSONL rows for both engines. Legacy vllm_metrics_sample / sglang_metrics_sample rows are translated on read, so old trace directories keep rendering. The sglang storage_* label prefix fold and bare host:port metrics URLs land in the next two commits. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…on unknown labels
sglang versions before upstream PR #37503 emit cache_source=f"storage_{backend}"
(e.g. "storage_HiCacheFile") instead of plain "storage". parseSGLangCounters now
folds any "storage"-prefixed label into the single "storage" key before the
delta-fold ever sees it, so a label-spelling change mid-fleet (rolling upgrade)
doesn't look like a brand-new series and break per-key continuity.
Also adds a one-time-per-process warning (not per-scrape) for any cache_source
value outside the known {device, host, storage*, total} vocabulary, so a future
unrecognized label doesn't silently vanish from every report band the way
storage_HiCacheFile did before this fix — its tokens are still parsed and
accumulated, just flagged as unmapped.
… a scheme A bare "host:port" (no "://") previously produced a schemeless URL that failed at request time with "unsupported protocol scheme", even though the --vllm-metrics-url/--sglang-metrics-url CLI help text has always claimed a bare endpoint is accepted. Now a bare host:port is assumed http:// before the rest of the /metrics-suffix normalization runs.
The sampler section still named the per-engine record types and their sources.* fields; samples are now cache_metrics_sample rows with bands.*. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
mixCat still switched on pre-unification band names, so the canonical gpu and external bands fell through to compute and GPU radix hits and KV-connector hits rendered as recomputation. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A generic spec probed for vLLM plus an explicit sglang URL writes both engines' cumulative samples into one file. buildSampleViz diffed them as a single series, so every engine switch read as a counter jump or reset. Difference each (model, engine) stream against its own previous sample, spread its delta over the intervals it spans, and sum the streams per band. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The unified sampler relabels vLLM local_cache_hit as the host band, which the 3D fill sent to DRAM. Each series now carries the engine whose host tokens dominate (hostEngine) and mixCat maps host to hbm for vllm, dram otherwise (sglang HiCache L2). The 2D labels and on-disk schema are unchanged. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A round where only some endpoints answered bumped consecFails but never reached noteSuccess, so a speculative sampler behind a partly reachable fleet never retired its budget, never logged that it serves the counter family, and was switched off by the first total outage after three partial rounds. A partial answer proves the endpoints serve metrics: it now counts as a success while EndpointsOK still records the coverage, and consecFails only counts rounds where nothing answered. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
7e98b8e to
da9825e
Compare
Follow-up to the sglang endpoint support already on
main: fixes cache-mix accounting bugs and unifies the vLLM/sglang cache-metrics sampler.Fixes
fold()treated any first-sighted counter key as a baseline, so a tier whose label appears mid-run (sglanghost/storageappear at their first nonzero hit) lost its accumulated value on an already-seen endpoint. The tier band was understated and compute overstated. A known endpoint's new key is now credited in full.storage_*cache_source labels are folded into onestorageseries (otherwise their tokens landed in compute), and unknowncache_sourcevalues warn once.host:portmetrics URLs (--vllm-metrics-url/--sglang-metrics-url) get anhttp://scheme instead of failing at request time.(model, engine)stream. Cumulative counters from different engines in one file were previously folded together, which produced wrong deltas when a run spanned vLLM and sglang.gpuandexternalwere missing from the 3D mapping. vLLM's local cache hits (reported ashost) are drawn as GPU cache rather than DRAM, since vLLM's local cache is the GPU prefix cache; sglanghoststays DRAM.--*-metrics-urlendpoints answer, the round is a success (so a spec that didn't saytype=openai_vllmno longer gives up after 3 partial rounds), and the "N of M endpoints answered" log appears once per streak.Refactor
metricsSamplerparameterized by a per-engine table replaces the separate vLLM and sglang samplers, with one report builder and one sharedmixBandStyleJS.cache_metrics_samplerows with canonical bands (compute,gpu/host/storage/external) and anenginefield. Legacyvllm_metrics_sample/sglang_metrics_samplerows are translated on read, so old trace directories still render.docs/replay.mddescribes the new rows.local_cache_hitis shown ashost, and vLLMexternalhas its own color, distinct from sglangstorage.Also
--sglang-metrics-urloverride end to end and the no-total-label shape (test:), plus regression tests for each fix above.Testing
task verify(gofmt,go vet,go test -race) passes on every commit in the series.🤖 Generated with Claude Code
Note
Medium Risk
Large refactor of benchmark metrics I/O and report embedding affects trace compatibility (mitigated by legacy JSONL translation) and cache-mix accuracy for fleet restarts and mixed engines.
Overview
Unifies vLLM and SGLang cache-metrics sampling into a single
metricsSamplerincache_metrics.go, with engine-specific parsing kept invllm_metrics.go/sglang_metrics.goviacacheEnginetables. Benchmark runs now writecache_metrics_sampleJSONL rows with canonicalbands(compute,gpu,host,storage,external); reports use onebuildSampleViz, sharedmixBandStyleJS, and read-time translation of legacyvllm_metrics_sample/sglang_metrics_samplefiles.Accounting and parsing fixes: delta
fold()now credits counter keys that appear mid-run on an already-scraped endpoint (fixes understated sglang host/storage and overstated compute). SGLangparseSGLangCountersfoldsstorage_*labels intostorageand warns once on unknowncache_sourcevalues.normalizeMetricsURLsprependshttp://for barehost:portoverride URLs.Report behavior: vLLM local cache maps to
host;externalgets a distinct color from sglangstorage. Mixed-engine traces diff per(model, engine)stream;hostEnginedrives 3D cache-mix categorization.readJSONLFileWithParamsreturns a single sample slice (signature change across callers/tests).Reviewed by Cursor Bugbot for commit da9825e. Bugbot is set up for automated code reviews on this repo. Configure here.