[Bug] DFlash + Qwen3.8-27B: speculative decoding accept rate drops to 0 during serving #43514

Checklist

• I searched related issues but found no solution.
• The bug persists in the latest version.
• Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback.
• If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed.
• Please use English. Otherwise, it will be closed.

Describe the bug

On a hybrid (GDN linear-attention + full-attention) model served with --speculative-algorithm DFLASH while grammar-constrained traffic (OpenAI tools + tool_choice: "required") is present, an entire running batch can suddenly become "poisoned":

• Every request in the batch — including plain-text requests with no grammar — starts emitting token id 0 ("!") repeatedly until max_tokens (the garbage never hits EOS/stop).
• Spec verify keeps executing (spec_verify_calls_total keeps increasing) but accept rate drops to exactly 0.00 / accept length 1.00.
• No error, no warning, no crash — outputs are silently corrupted.
• The state persists for each in-flight request's lifetime; newly admitted requests are healthy again once the triggering condition leaves the batch.
• The transition is a one-step collapse (production trace below: accept 0.48 -> 0.17 -> 0.06 -> 0.00 in three decode steps).

Two production incidents observed:

1. v0.5.20, one instance of three, poisoned for ~5 days until we rolled back (roughly 12% of traffic emitting garbage).
2. v0.5.19, one instance of four, poisoned for ~1.5 h and self-recovered when traffic mix changed. So this is not a v0.5.20 regression — both tags are affected.

Reproduction

Semi-reliable (~30% per round on an instance that has recently served real mixed traffic; 0/10 on a freshly started instance). Model: a pre-release Qwen3.8-27B hybrid (GDN + full attention) with its DFlash2 draft model; launch flags:

--speculative-algorithm DFLASH --speculative-num-draft-tokens 6 --enable-mixed-chunk \
--enable-strict-thinking --enable-hierarchical-cache --mamba-radix-cache-strategy extra_buffer \
--attention-backend flashinfer --tp 4

Per round (run 10 rounds, fresh content each round):

# 1) 10 grammar requests (tool_choice=required), max_tokens=600, temperature=1.0
for v in 0 1 2 3 4; do send g$v.json & send g$v.json & done   # g*.json: tool-call requests
sleep 1
# 2) 30 concurrent requests with FRESH ~50K-token prompts (multi-chunk prefills)
for i in $(seq 30); do send flood.json & done
sleep 3
# 3) plain probe (max_tokens=40), inspect output text

Observed (two consecutive runs on the same instance):

run 1: rounds 3/6/8 of 10 -> probe output '!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!'
run 2: rounds 3/5   of 10 -> probe output '!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!'
control on a freshly started instance: 0/10

Controls: grammar requests alone + probe (no flood) -> healthy; flood alone (no grammar) -> healthy. Both ingredients are required: grammar requests in the batch and multi-chunk admissions reshaping the batch.

Production trigger (log reconstruction of the Oct 10 incident, v0.5.19):

10:05:35  healthy decode: run=5, accept_len=3.38, accept_rate=0.48   (pools idle: token_usage=0.26, no retraction)
10:05:41  two admissions in the same second:
            newseq=1 new-token=8128 cached=155904   (long-prompt chunk tail)
            newseq=3 new-token=8128 cached=129792   (three requests join at once; run 5->8)
10:05:42  next decode: accept_len 3.38->1.87, accept_rate 0.48->0.17   (one-step collapse)
10:05:43  accept_rate 0.06; first victim finishes (prompt=78634, completion=62, accept_rate=0.0)
10:05:44  accept_len=1.00, accept_rate=0.00 -- stays there for ~1.5 h

Just before the trigger, several long-prompt / short-completion (~100-300 token output, tool-call-shaped) requests had finished — i.e. grammar-bearing requests left the batch seconds before the batch was reshaped.

Profiling captured live during a poisoned window:

• py-spy stack: build_grammar_vocab_mask -> generate_token_bitmask -> traverse_tree/dfs inside forward_batch_generation (the DFlash verify path).
• Flame graph (15 s of the poisoned verify loop): spec_utils.dfs = 57% of scheduler CPU, build_grammar_vocab_mask = 30%; the accept kernels (accept_sampling / _accept_block) < 1%.

Environment

(Reconstructed from the incident deployment — direct check_env run is restricted on that host.)

- sglang: v0.5.19 and v0.5.20 (both affected)
- Python: 3.12.3
- GPU: 4x NVIDIA H20 96GB, driver 580.95.05, CUDA 13.0
- attention backend: flashinfer; TP=4
- model: hybrid GDN linear-attention + full-attention (~27B) + DFlash2 draft
- overlap scheduling enabled (default)