What happened
MTP draft acceptance dropped from 33% to 4% on the identical machine, model, prompt, and configuration between pre-#68 main (commit 1bdaeee) and current dev HEAD (6d3ed7e). At 4% the adaptive guard disables drafts after 24 proposals, so speculation effectively never engages anymore.
A/B data
GLM-5.2 744B int4 + int8 MTP head (mateogrgic checkpoint), Ryzen 9 9950X3D, 126 GB, Windows 11 native (MinGW GCC 16.1). Same prompt ("Summarize the plot of Romeo and Juliet in three sentences."), same warm .coli_usage, --ram 75 --ngen 32 --topp 0.7, 40 GB pin, MLOCK=1:
| engine |
speculation |
acceptance |
| main @ 1bdaeee (pre-#68) |
2.00 tok/forward (16 fw / 32 tok) |
33% (16/48) |
| dev @ 6d3ed7e |
1.07 tok/forward (30 fw / 32 tok) |
4% (1/24) → auto-disabled |
Reproduced at temp 0.7 and at --temp 0 (greedy), and identically with DIRECT=0 vs DIRECT=1 (byte-identical greedy outputs across the I/O paths, so it's not a data/read issue). On the older engine the same head reached 56–83% acceptance on other prompts; the 33% row is the strict same-config comparison.
Possibly related: #156 reports MTP as a net loss on Windows (DRAFT=0 faster) — if that machine ran a post-#68 build, this regression rather than cold-cache draft cost may be the real cause.
Suspects
The #68 decode-batching rework touched the draft/verify path (spec_decode/run_serve_mux area). Acceptance collapsing by ~8× with the same int8 head suggests the verification distribution or the draft context changed, not the head quality. One more observation from the same runs, possibly the same root cause: greedy decode at --topp 0.7 now opens the R&J summary with off-distribution text ("Aegeanin (assuming you meant …") — deterministic and identical across I/O paths.
Happy to run bisects or instrumented builds on this rig (it's fast to iterate: warm cache, ~60 s per 32-token run).
🤖 Generated with Claude Code
What happened
MTP draft acceptance dropped from 33% to 4% on the identical machine, model, prompt, and configuration between pre-#68
main(commit 1bdaeee) and currentdevHEAD (6d3ed7e). At 4% the adaptive guard disables drafts after 24 proposals, so speculation effectively never engages anymore.A/B data
GLM-5.2 744B int4 + int8 MTP head (mateogrgic checkpoint), Ryzen 9 9950X3D, 126 GB, Windows 11 native (MinGW GCC 16.1). Same prompt ("Summarize the plot of Romeo and Juliet in three sentences."), same warm
.coli_usage,--ram 75 --ngen 32 --topp 0.7, 40 GB pin, MLOCK=1:Reproduced at temp 0.7 and at
--temp 0(greedy), and identically withDIRECT=0vsDIRECT=1(byte-identical greedy outputs across the I/O paths, so it's not a data/read issue). On the older engine the same head reached 56–83% acceptance on other prompts; the 33% row is the strict same-config comparison.Possibly related: #156 reports MTP as a net loss on Windows (DRAFT=0 faster) — if that machine ran a post-#68 build, this regression rather than cold-cache draft cost may be the real cause.
Suspects
The #68 decode-batching rework touched the draft/verify path (
spec_decode/run_serve_muxarea). Acceptance collapsing by ~8× with the same int8 head suggests the verification distribution or the draft context changed, not the head quality. One more observation from the same runs, possibly the same root cause: greedy decode at--topp 0.7now opens the R&J summary with off-distribution text ("Aegeanin (assuming you meant …") — deterministic and identical across I/O paths.Happy to run bisects or instrumented builds on this rig (it's fast to iterate: warm cache, ~60 s per 32-token run).
🤖 Generated with Claude Code