Developing my own LLM Inference server like vLLM. To understand the core concepts deeply by implementing them from scratch.

1. Online and Offline mode.
2. KV Caching
3. Multiple requests processing.
4. Requests scheduler.
5. Seperate prefill and Decode.
6. Prefix caching
7. Continuous batching
8. Chunked prefill
9. Torch compilation
10. CUDA Graphs
11. Distributed inference (TP)

1. Speculative decoding
2. Quantization

While doing the developement of each of above major techniques. I compared the tokn with vLLM and logged the numbers for Throughtput, TTFT, TPOT, ITL etc.

My first log after doing some core techniques implementation. I randomly benchmarked tokn with vLLM on 102 prompt.

From this point on I generated proper synthetic dataset to benchmark tokn properly with vLLM. 400 prompts (mixed lengths), Qwen3-0.6B, bf16, max_tokens=256, max_model_len=2048, greedy decoding.

• prefix caching
• chunked prefill.

vLLM is 1.41× tokn throughput.

• Continuous batching

vLLM is 1.48× tokn throughput.

• torch.compile

vLLM is 1.67x tokn throughput.

TTFT = time to first token, TPOT = time per output token, ITL = inter-token latency.

• CUDA Graphs

vLLM is 1.02× tokn throughput.