GLM-5.3-Flash · DFlash2 · TP4 · 1M Context

_{by Tony Liu — 4× DGX Spark home cluster}

_{The 102 tok/s badge is an upper bound measured with reasoning suppressed and a maximally predictable prompt — see Benchmark protocol. A default client on a natural prompt sees roughly 30–39 tok/s.}

DFlash2 speculative decoding for zai-org/GLM-5.3-Flash at tensor-parallel 4 across four NVIDIA DGX Spark, serving the model's full 1,048,576-token context.

The DFlash2 drafter and the vLLM overlay that makes it run on GB10 are tonyd2wild's, who also run it at TP4. Two things here are different:

A single-variable comparison against MTP-4. Same nodes, same context, same KV pool, same batch budget, only --speculative-config changes — across four content types and a full vllm bench serve latency-throughput curve, so the gain can be attributed and its limits stated rather than asserted.

--max-num-batched-tokens. Left unset, vLLM derives 2048 from the speculative settings and warns that this is suboptimal. Raising it to 8192 is worth −29 % TTFT and +42 % prefill throughput for +1 GiB; the ladder is below.

For KV-pool capacity that recipe is ahead — 5,033,164 tokens fp8 at 32 GiB/rank, and 6,652,112 with the NVFP4 KV lane it ports from drowzeys. This runs 4,545,221 at 28 GiB/rank, which is where head-rank memory tops out on our fleet.

Two NVFP4 conversions of the same base model do not behave the same. The ModelOpt-produced ones emit invalid UTF-8 byte-token sequences; a compressed-tensors conversion of the same model, on the same image, kernel, KV layout, parallelism and sampling settings, does not. Reported upstream as vllm#54150 on SM120. It reproduces on SM121.

The failure is quiet. The server returns HTTP 200, throughput looks normal, and English text is mostly fine — it surfaces as U+FFFD inside multi-byte characters, which reads like a client or font problem.

Measured here, 24 generations per column, max_tokens 3000, temperature 0 and 0.7 in equal halves, prompts chosen to force CJK and emoji output:

Corruption samples from the ModelOpt checkpoint — note that a lost character can leave an ASCII fragment behind, so this is byte-level and not a rendering artefact:

取得<?>面上可互動元素      ("畫" lost mid-word)
| <?>Egg **溏心蛋** |       (emoji collapsed into an ASCII fragment)

On our own gateway logs over four days, 16 of 1,997 real requests (0.80 %) contained U+FFFD in the response while the prompt contained none — so this is not a benchmark artefact, it is a live defect at roughly one request in 125.

Two practical notes on switching:

• The compressed-tensors checkpoint is 197.9 GB in 21 files and ships an MTP head (model_mtp.safetensors, 7.62 GB, model.language_model.layers.45.* with eh_proj/enorm/hnorm) that the ModelOpt one does not. It is referenced by model.safetensors.index.json, so copy the whole directory — do not cherry-pick the ten shards. Running DFlash2 it is simply not instantiated; loading is warning-free.
• Everything else is identical. Both config.json files declare the same num_hidden_layers: 45, num_nextn_predict_layers: 1, the same 34 KDA / 11 DSA layer_types, the same mlp_layer_types. No launcher flag changes, and the KV pool is bit-identical at 4,545,221 tokens.

If you want to check your own deployment, the cheapest test is a long Korean generation — that was the most sensitive probe here by a wide margin. Counting U+FFFD in the response while asserting the prompt has none separates model-side corruption from echoed input.

Two regimes, and they differ enough that a single headline number would mislead.

Four content types, three runs each, median. Prompts from MiaAI-Lab/sparkDash so the figures are comparable to that project.

Both columns were measured on the ModelOpt quant, before the swap described above; the comparison is single-variable in --speculative-config, which is what it is for. What the quant swap itself did to these numbers is at the end of this section.

ms/step is 76.1–80.3 → 77.9–80.2, i.e. unchanged. DFlash2 is a block-diffusion drafter: its 7 positions are produced in one parallel pass, not seven autoregressive ones. The step gets no longer, it just carries more accepted tokens — 2.9–5.0 → 5.38 tokens per step.

Same launcher, same flags, same effective page size — only the checkpoint changed. Three boots on the compressed-tensors quant against the one ModelOpt boot the table above reports:

The honest headline is no regression, with json and structured clearly ahead. We are not claiming the rest, and the reason is in that table: two boots of the identical configuration differ by up to 12 % (code 112.3 vs 100.7, prose 48.7 vs 42.7). Every boot recaptures CUDA graphs and lands a different memory layout, and bench-v3's three repetitions all live inside one boot, so they average away request-level noise and leave boot-level noise untouched.

If a configuration change requires a restart, the repetitions have to be at the boot level, not the request level. Ours were not, which is why two of these four rows stay unresolved rather than being reported as a gain. Acceptance rate is the exception worth trusting here: it clusters tightly across boots because it is a property of the weights, not of the allocator.

The single-stream table is an upper bound under a stated protocol, not what a default client sees. Two of the settings matter enough to spell out:

temperature: 0, top_p: 1                              # sparkDash protocol
chat_template_kwargs: {"reasoning_effort": "low"}     # OUR addition — not sparkDash's

GLM-5.3-Flash reasons by default. Suppressing that is our own deviation from the sparkDash protocol, and it is the single largest reason the headline number is what it is. The code prompt is also deliberately maximally predictable — 50 functions of identical shape, no comments, no docstrings — which is what drives 99.2 % acceptance.

Decomposed on the same engine, same day, idle cluster, max_tokens: 2000:

Prompt shape costs 38 % (99.3 → 61.6). Reasoning costs a further 37 % (61.6 → 38.7). Together, −61 %. On the benchmark prompt alone, reasoning looks nearly free (−8 %) — because that prompt is rigid enough that the model barely reasons at all. That is precisely why the headline number is fragile: it sidesteps both effects at once.

A default client on a natural prompt sees roughly 30–39 tok/s. Our production monitoring, which observes real gateway traffic rather than benchmark runs, reports TPOT p50 in the same region. Both figures are correct; they describe different populations.

One thing neither number captures: reasoning arrives before content, and there can be a lot of it. On a natural coding request with max_tokens: 2500, the entire budget went to reasoning — time-to-first-token was 304 ms, but the budget ran out at 89 s with the first content token still to come. The model was not failing; it simply had not finished thinking. Raise the budget and the answer arrives. If you are sizing for felt latency rather than throughput, measure time-to-first-content separately — TTFT will not tell you.

vllm bench serve, sonnet dataset, Poisson arrival, --ignore-eos, p50/p90/p99.

Throughput lands within ±2.4 % at every rate the server can actually sustain. The QPS 1.0 row is the median of three runs. Saturation is at 1.0 QPS for both methods — past that, throughput gains ~25 % while TTFT p90 grows an order of magnitude, so 1.0 QPS is the number to size capacity against.

Cold 32,768-token prompt, single stream:

Decode is unaffected by the drafter at depth. The TTFT difference is scheduling on the prefill side.

Betting 7 positions in parallel pays in full when the continuation is predictable and returns little when it is not. If your traffic is mostly free-form prose, MTP-4 keeps more of its drafts. If it is code, structured output or tool calls, DFlash2 is the better bet by a wide margin.

The pool grows because the drafter's five sliding-window layers slot-share the MLA tensors the way GLM's own mamba layers and kpool tails already do, and it replaces the MTP head that was there before. Per-block cost is bit-identical to the drafterless model.

4× GB10 (SM121, 121 GiB unified memory each), CX-7 RoCE, tensor-parallel 4, vLLM native multi-node (no Ray).

--tensor-parallel-size 4
--max-model-len 1048576
--max-num-batched-tokens 8192
--max-num-seqs 32
--block-size 2304
--moe-backend marlin
--kv-cache-dtype fp8_e4m3
--kv-cache-memory 30064771072
--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}'
--speculative-config '{"method":"dflash","model":"/models/dflash2-draft","num_speculative_tokens":7}'

num_speculative_tokens is 7, not 8: the drafter is trained for a block of 8 and one position is the target's own verified token.

--block-size is not a free knob on this model, and it fails silently. GLM-5.3-Flash is a hybrid: 34 KDA linear-attention layers alongside 11 sparse-attention ones. vLLM sizes the attention page against the mamba page and raises anything smaller, at INFO level, with no warning:

[interface.py:926] Setting attention block size to 2304 tokens to ensure that
                   attention page size is >= mamba page size.
[interface.py:950] Padding mamba page size by 0.70% to ensure that mamba page size
                   and attention page size are exactly equal.

We passed --block-size 576 and rebooted the fleet to see whether a finer page would buy finer prefix-cache reuse. It bought nothing, because the flag never took effect — the value on the command line and the value in force differ, and only the log says so. Two boots at 576 and one at 2304 were, in fact, three boots of the same configuration.

Read the effective value back out of the engine log before attributing anything to a vLLM flag. The prefix-cache consequence is in the next section.

--max-num-batched-tokens deserves its own note. Leave it unset and vLLM derives 2048 from the speculative settings, then warns that this is suboptimal. Raising it, measured on a 32K prompt:

−29 % TTFT and +42 % prefill throughput for +1 GiB. The slope flattens after 8192 while head-rank headroom drops to 9 GiB, which is where this stops.

A consequence of the page-size floor above, and it is worth knowing before you size a multi-turn agent against this model. Reuse does not land on the true common prefix; it lands on the last whole 2,304-token page minus one:

cached_tokens = max(0, (floor(prompt_tokens / 2304) - 1) * 2304)

Measured on an idle engine, same prompt twice, reading the delta of vllm:prefix_cache_hits_total and vllm:prefix_cache_queries_total:

Seven of seven. Two consequences:

• A prompt under 4,608 tokens gets no reuse at all. Not "a little" — zero. Short agent turns, tool-call round-trips and chat replies all repay full prefill every time.
• Every request re-computes between 2,304 and 4,607 tokens no matter how long the conversation is. At the prefill rates in the ladder below that is roughly 1–2 s of TTFT you cannot cache away.

--prefix-match-unit does not help: vLLM disables fine-grained lookup outright because KpoolTailManager and SlidingWindowManager require block-aligned lookups, and logs it at kv_cache_coordinator.py. Hit curves with and without the flag are line-for-line identical. --block-size does not help either, for the reason in the previous section. This is structural on hybrid mamba/attention models, not a tuning mistake.

No image build required — the runtime image is public and anonymous-pullable.

# one node pulls, then fan out over the internal fabric
docker pull ghcr.io/tonyd2wild/vllm-glm53-flash:sm121-v11-dflash2
docker save ghcr.io/tonyd2wild/vllm-glm53-flash:sm121-v11-dflash2 | zstd -3 -T8 \
  | ssh <peer> "zstd -d | docker load"

Pull from one node only. Four nodes pulling the same 31 GB image concurrently hits GHCR rate limits and the last layers retry indefinitely; the internal 200 Gb fabric moves it in about four minutes per node.

# weights
#   target  : RedHatAI/GLM-5.3-Flash-NVFP4     -> ~/dev/models/GLM-5.3-Flash-NVFP4
#             compressed-tensors, NOT the ModelOpt quants -- see "The quant is not
#             interchangeable" below. 197.9 GB, 21 files, ships an MTP head as well.
#   drafter : incoai/GLM-5.3-Flash-DFlash2     -> ~/dev/models/GLM-5.3-Flash-DFlash2  (2.2 GB)

# patches
mkdir -p ~/glm53-patches && cp patches/*.py ~/glm53-patches/
cp patches/kv_cache_utils.py ~/glm53-patches/kv_cache_utils_dflash2.py
cp scripts/launch-glm53-dflash2-tp4.sh scripts/cache_flusher.sh ~/

# every node: unconditional page-cache flusher for the whole boot, workers first
nohup bash ~/cache_flusher.sh &
bash ~/launch-glm53-dflash2-tp4.sh 3     # worker
bash ~/launch-glm53-dflash2-tp4.sh 2     # worker
bash ~/launch-glm53-dflash2-tp4.sh 1     # worker
bash ~/launch-glm53-dflash2-tp4.sh 0     # head — serves :8000

On GB10 the CPU and GPU share one 121 GB pool, so host page cache consumes CUDA-visible memory one-for-one. The flusher must run unconditionally — a threshold-triggered one can sit below its threshold and still leave the allocator short, which shows up as the same command booting or OOMing depending on the moment.

Edit the rank→IP map at the top of the launcher. Ours: RoCE on 10.10.0.x, HCA rocep1s0f1, NCCL_IB_GID_INDEX=3, RoCEv2.

sim_glm5_drafter.py from the upstream overlay drives the patched kv_cache_utils on CPU with the real spec geometry — 34 mamba + 11 MLA + 11 indexer + 11 tail + 5 drafter SWA — and asserts the tensor binding stays in bounds under the runner's kernel-block split. Run it against this repo's merged file before touching a node:

[A] drafter: block 2048->2304, real page == mla_page, no padding
[A] bytes/block unchanged by drafter: 26155008
[A] runner reshape OK under kernel split 2304->64 (x36):
    required 398721024 B == tensor 398721024 B
ALL SIMULATION CHECKS PASSED

The drafter's rescaled block lands exactly on --block-size 2304, so it takes the exact-fit path with no page padding.

Using Eagle3 auxiliary layers from config: (6, 15, 25, 34, 43)
GPU KV cache size: 4,545,221 tokens, Maximum concurrency for 1,048,576 tokens per request: 4.33x
Warming up spec-decode rejection sampler kernels (vocab=154880, num_spec=7, ...)

Acceptance from /metrics (spec_decode_num_accepted_tokens_total ÷ ..._num_draft_tokens_total) settles between 0.5 and 0.65 on mixed traffic. A ratio near 0.15 means the auxiliary capture or the mHC contraction is wrong — it degrades silently rather than raising. Read it only after a few thousand drafts have accumulated; early samples swing wildly (34 % at 90 drafts, 56 % at 16k, 62 % at 22k on the same deployment).

patches/kv_cache_utils.py carries two independent changes to one file:

They touch different functions and compose cleanly. The drafter's bytes flow into the null-block accounting automatically, which is the behaviour you want.

The other three files in patches/ route NoPE MLA away from the packed fp8_ds_mla layout and add a feasibility fallback to the kpool indexer's persistent_topk.

Four GB10 Sparks (SM121, 121 GiB unified memory each), CX-7 RoCE on 10.10.0.0/24 through a MikroTik CRS812. Boot ≈ 20 min: weight load, drafter load, KV allocation, graph capture.

Model zai-org/GLM-5.3-Flash · quant RedHatAI/GLM-5.3-Flash-NVFP4 (compressed-tensors) · DFlash2 drafter incoai/GLM-5.3-Flash-DFlash2 · the DFlash2 vLLM overlay, the drafter-group KV patch, the simulator and the runtime image are tonyd2wild's — this repo is the TP4 / 1M carry and the measurement · NVFP4 KV lane and the GB10 memory study from drowzeys · decode benchmark prompts from MiaAI-Lab/sparkDash · vLLM PR #53906 authors for the day-0 image · vllm#52520 for the null-block diagnosis.

MIT.