LvLLM = vLLM + lk_moe, plus SM80/86/89 adaptation and SM120 tuning/fixes for popular new models (DeepSeek-V4/V4.1, Qwen3.8, GLM-5.3, …).
- lk_moe is the CPU+GPU hybrid, NUMA-aware MoE engine (pip-installable); vLLM provides the GPU path. LvLLM is lk_moe's integration into vLLM plus the low-arch (SM80/86/89) bring-up and SM120 fixes for these models.
- Fully optional: with
LVLLM_MOE_NUMA_ENABLED=0it behaves exactly like stock vLLM.
| Model | SM80 | SM86 | SM89 | SM90 | SM100 | SM120 | spec-decode |
|---|---|---|---|---|---|---|---|
| DeepSeek-V4.1-Flash | ✅ new | ✅ new | ✅ new | ✅ native | ✅ native | ✅ fixed | ✅ dspark |
| DeepSeek-V4-Flash (0731) | ✅ new | ✅ new | ✅ new | ✅ native | ✅ native | ✅ native | ✅ dspark |
| Qwen3.8-Flash-Next | ✅ new | ✅ new | ✅ new | ✅ native | ✅ native | ✅ new | ✅ MTP |
| GLM-5.3-Flash | ✅ new | ✅ new | ✅ new | ✅ native | ✅ native | ✅ fixed | ✅ MTP |
native = upstream vLLM · new = support added by this Lvllm release · fixed = upstream path
corrected by this Lvllm release. Full per-model hardware tables, benchmarks and CLI:
RELEASE_NOTES.md.
Original MOE models verified on vLLM's Qwen3 / GLM / MiniMax lines:
| Family | Models |
|---|---|
| Qwen3 | 3.6 / 3.5-35B-A3B, 3.5-122B-A10B, 3.5-397B-A17B, Coder-Next, Next-80B-A3B-Instruct, Coder-30B-A3B-Instruct, VL-30B-A3B-Instruct |
| GLM | GLM-5.2-NVFP4 [sm120], GLM-4.7(-Flash) / 4.6V |
| MiniMax | M3 / M2.7 / M2.5 / M2.1 |
| Others | gemma-4-26B-A4B-it, NVIDIA-Nemotron-3-Super-120B-A12B-BF16, Ornith-1.0-35B-FP8, Kimi k2.6 / k2.5 |
Unlisted original MOE models from the Qwen3 / GLM / MiniMax lines are theoretically supported, pending testing.
Single request, greedy decode (t/s). Each cell = plain / spec t/s; spec = dspark for
DeepSeek-V4.x, MTP for Qwen3.8. — = not applicable. Full hardware tables: RELEASE_NOTES.md.
| Model | SM120 TP2 (2× 5060 Ti) | SM86 TP2 (2× 3090) | Mixed TP4 (3090×2 + 5060 Ti×2) |
|---|---|---|---|
| DeepSeek-V4.1-Flash | 25 / 32–37 | 27 / 26–40 | 22 / 24–34 |
| DeepSeek-V4-Flash (0731) | 28 / up to 44.8 | 30.6–31.1 / 36–43 | 30.5–33.6 / 38–43.5 |
| Qwen3.8-Flash-Next | — | 45 / — | 40.3–40.6 / up to 78 |
| GLM-5.3-Flash | 20.8 / — | 22–23 / — | 22.4 / — |
All numbers above were measured on one host:
| Item | Spec |
|---|---|
| CPU | 2× AMD EPYC 7642 48-Core (96c/192t total, NPS4 ⇒ 8 NUMA nodes), LK_THREADS=48 |
| Host RAM | 1 TiB DDR4-3200 (16-channel, 8 per socket) |
| RAM headroom | DeepSeek-V4.x peak ≈ 590 GB resident (≥ 640 GB advised); Qwen3.8 peak ≈ 335 GB |
lk_moe spans a MoE model across VRAM + system memory and schedules experts across CPU + GPU with NUMA awareness — reaching ~100% VRAM utilization and overlapping GPU prefill with hybrid decode.
| Role (per MoE layer) | Meaning | Env |
|---|---|---|
| master switch | 0 = stock vLLM pure-GPU, 1 = hybrid |
LVLLM_MOE_NUMA_ENABLED |
| GPU-prefill layer | big batches on GPU, small on CPU | LVLLM_GPU_PREFILL_MIN_BATCH_SIZE + LVLLM_GPU_PREFETCH_WINDOW |
| GPU-resident layer | weights stay in VRAM | LVLLM_GPU_RESIDENT_MOE_LAYERS |
Requires x86 AVX2+ and an NVIDIA GPU (SM75+).
Ready-made serve scripts (per model × topology) live in commands/:
commands/
dsv4_0731_serve_tp2_{3090,5060ti}[_dspark].sh dsv4_0731_serve_tp4[_dspark].sh # DeepSeek-V4-Flash 0731
dsv41_serve_tp2_3090_dspark.sh # DeepSeek-V4.1-Flash
qwen38_serve_tp2_{3090,5060ti}[_mtp].sh qwen38_serve_tp4[_mtp].sh # Qwen3.8-Flash-Next
glm53_serve_tp2_{3090,5060ti}[_plain].sh glm53_serve_tp4.sh # GLM-5.3-Flash (--kv-cache-dtype bfloat16 on sm8x)
Each script is a complete, self-contained vllm serve … (env + args). Pick by model and GPU
topology, adjust the model path, and run:
bash commands/dsv4_0731_serve_tp4_dspark.sh| Env var | Default | Description |
|---|---|---|
LVLLM_MOE_NUMA_ENABLED |
0 |
hybrid on/off (0 = stock vLLM) |
LK_THREADS |
— | threads = physical cores ÷ #GPUs |
LK_THREAD_BINDING |
CPU_CORE |
CPU_CORE (best) / NUMA_NODE |
LVLLM_GPU_PREFETCH_WINDOW |
— | prefetch window, typically 1 |
LVLLM_GPU_PREFILL_MIN_BATCH_SIZE |
— | GPU prefill starts at input ≥ value; 0 = off |
LVLLM_GPU_RESIDENT_MOE_LAYERS |
none | expert layers in VRAM, e.g. 0, 0-1,9 |
LVLLM_ENABLE_NUMA_INTERLEAVE |
1 |
avoid NUMA node OOM |
LK_POWER_SAVING |
0 |
1 = CPU power saving |
VLLM_DSV4_UNPACK_INDEXER |
0 |
DeepSeek-V4.x: carve indexer pages out of the interleaved KV pool (mixed-arch TP prefix-reuse crash workaround) |
On a mixed-arch host add FLASHINFER_CUDA_ARCH_LIST="<all ranks>" (e.g. "8.6 12.0f").
- Output performance:
--compilation-config '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_DECODE_ONLY"}'. - Enable GPU prefill:
LVLLM_GPU_PREFETCH_WINDOW=1,LVLLM_GPU_PREFILL_MIN_BATCH_SIZE=4096,--max-num-batched-tokens 32000; disable with…MIN_BATCH_SIZE=0+--max-num-batched-tokens 4096. - Heterogeneous TP (>2 PCIe GPUs): add
--disable-custom-all-reduce. - BIOS NUMA: AMD EPYC NPS4 / Intel SNC4, node count a multiple of GPU count.
pip install https://github.com/guqiong96/Lvllm/releases/download/lvllm-v2.5.1/lvllm-2.5.1-cp312-cp312-manylinux_2_34_x86_64.whl
# deps (incl. the bundled FlashInfer wheel with the SM120 sparse-MLA fix) resolve automaticallyFrom source (needs GCC ≥ 13 for <format>):
git clone https://github.com/guqiong96/Lvllm.git && cd Lvllm
for d in /opt/rh/gcc-toolset-{14,13}/enable; do [ -f "$d" ] && source "$d" && break; done # RHEL/Rocky
VLLM_VERSION_OVERRIDE="2.5.1" CMAKE_BUILD_TYPE=Release pip install -e . --no-build-isolation- In this repo: only the lk_moe integration —
patches/01_lk_moe__71888f507a.patch, a single diff of the current tree over upstream base71888f507a.git checkout 71888f507a && git apply patches/01_lk_moe__71888f507a.patch - The per-model / PR patches are not shipped here; they live in
~/Downloads/opencode/patches/.