Version: sglang 0.5.21 (lmsysorg/sglang:v0.5.21-cu130), mooncake transfer backend, TP 1, one node.
Model: Qwen/Qwen3.5-0.8B (hybrid linear and full attention). Every engine runs with
--max-running-requests 256 --page-size 1 --attention-backend triton --enable-pd-role-switch --disaggregation-transfer-backend mooncake.
Steps
- Start engine A with
--disaggregation-mode prefill and engine B with --disaggregation-mode decode, each with its own --disaggregation-bootstrap-port.
- Serve requests A -> B: they work.
- With both engines idle,
POST /pd_role_switch {"new_role": "decode", "decode_cuda_graph_memory_gb": <B's value from /server_info>} on A, then {"new_role": "prefill"} on B. Serve B -> A: it works.
- Switch A back to prefill and B back to decode (passing
decode_cuda_graph_memory_gb). Both switches return success: true.
- Serve A -> B.
Expected: B decodes as it did in step 2.
Actual: B's scheduler exits on its first decode with torch.AcceleratorError: CUDA error: an illegal memory access was encountered, then SIGQUIT. Prefill requests fail after the bootstrap timeout. This reproduced in 4 of 4 runs, including with a larger model from the same family.
Workaround: launch every engine with --disaggregation-mode prefill and switch the decode engines to decode at startup. With that, engine A and engine B each switched both ways through three full cycles and served every request (2 of 2 runs). So the failure seems tied to an instance whose launch role was decode.
Version: sglang 0.5.21 (
lmsysorg/sglang:v0.5.21-cu130), mooncake transfer backend, TP 1, one node.Model: Qwen/Qwen3.5-0.8B (hybrid linear and full attention). Every engine runs with
--max-running-requests 256 --page-size 1 --attention-backend triton --enable-pd-role-switch --disaggregation-transfer-backend mooncake.Steps
--disaggregation-mode prefilland engine B with--disaggregation-mode decode, each with its own--disaggregation-bootstrap-port.POST /pd_role_switch {"new_role": "decode", "decode_cuda_graph_memory_gb": <B's value from /server_info>}on A, then{"new_role": "prefill"}on B. Serve B -> A: it works.decode_cuda_graph_memory_gb). Both switches returnsuccess: true.Expected: B decodes as it did in step 2.
Actual: B's scheduler exits on its first decode with
torch.AcceleratorError: CUDA error: an illegal memory access was encountered, then SIGQUIT. Prefill requests fail after the bootstrap timeout. This reproduced in 4 of 4 runs, including with a larger model from the same family.Workaround: launch every engine with
--disaggregation-mode prefilland switch the decode engines to decode at startup. With that, engine A and engine B each switched both ways through three full cycles and served every request (2 of 2 runs). So the failure seems tied to an instance whose launch role was decode.