English | 中文

A lightweight and optimized vLLM implementation built on Nano-vLLM and OpenAI Triton.

This project extends the original Nano-vLLM with the following improvements:

1. Extended Model Support: Added comprehensive support for:

   • Qwen2 series models
   • Qwen2.5 series models
   • Qwen3-MoE series models
   • Llama series models
   • Original Qwen3 series models

2. Triton Optimization: Implemented custom Triton operators:

   • softmax_online: Replaces torch.softmax
   • cat_cos_sin: Replaces torch.cat(cos, sin, dim=-1)
   • Performance Gain: Average speed improvement of 101.93 tokens/s

git clone https://github.com/cosmoliu2002/nano-vllm-triton.git
cd nano-vllm-triton
pip install -e .

example.py

python example.py --model-path /path/to/your/model

Supported Parameters

--model-path
--tensor-parallel-size
--enforce-eager
--temperature
--max-tokens

run_example.sh

bash run_example.sh

Supported Parameters config_example.yaml

model_path: "/path/to/your/model"
tensor_parallel_size: 1
enforce_eager: true
temperature: 0.6
max_tokens: 256

run_bench.sh

bash run_bench.sh

Supported Parameters config_bench.yaml

model_path: "/path/to/your/model"
engine: "nanovllmtriton"             # nanovllmtriton or vllm

Test Configuration:

• Hardware: NVIDIA A10 (Powered by ModelScope Notebook)
• Models: Various Qwen and Llama models
• Total Requests: 256 sequences
• Input Length: Randomly sampled between 100–1024 tokens
• Output Length: Randomly sampled between 100–1024 tokens
• Test Runs: 5 runs per model with average results

Test Results:

• Qwen Series: Qwen2, Qwen2.5, Qwen3, Qwen3-MoE
• Llama Series: Llama 2, Llama 3, Llama 3.1, Llama 3.2

Gradually replace all PyTorch implementations with custom Triton kernels

• Nano-vLLM
• Qwen2, Qwen2.5, Qwen3-MoE, Llama series model support Nano-vLLM-gogongxt
• OpenAI Triton