wLLM — The Windows Native Inference Engine

wLLM is a 100% ground-up, high-performance inference engine specifically architected for the Windows ecosystem. Built in pure Python and PyTorch, it delivers server-grade continuous batching and KV-paging capabilities to consumer hardware without the Linux-dependency overhead of vLLM.

wLLM uses a dynamic profiling engine to mathematically tune its performance based on your exact hardware signature. No static profiles, just raw math.

Unlike standard sequential servers, wLLM treats the GPU as a shared resource that accepts new requests into the batch at every token iteration.

graph TD
    Client([API Client]) -->|v1/chat/completions| API[FastAPI Server]
    API -->|Submit Request| S[Scheduler Queue]
    
    subgraph Engine [The Inference Core]
        S -->|Admit| Loop[Inference Loop Thread]
        Loop -->|Check Budget| KV[KV Cache Manager]
        KV -->|Allocate Blocks| B[Paged Memory Blocks]
        Loop -->|Process Iteration| E[Batched Generation Step]
        E -->|Generate| Sampler[Logit Sampler]
        Sampler -->|Token Output| Loop
    end
    
    Loop -->|Stream| API
    API -->|SSE Stream| Client

1. Zero-Day Architecture Support: Any AutoModelForCausalLM on HuggingFace works instantly. No waiting for community GGUF conversions.
2. Infinitely Extensible: 100% Python. Modify scheduling logic, integrate custom logit processors, or add backends without touching a C++ compiler.
3. Structured Output (JSON mode): OpenAI-style response_format with grammar-constrained decoding via xgrammar — guaranteed-valid JSON from any model, even zero-day ones (pip install winllm[structured]).
4. CUDA Graph Decode: --cuda-graphs captures the decode forward in a CUDA graph (no Triton/torch.compile needed on Windows) — up to ~7× single-request decode throughput, with an automatic eager fallback for graph-unsafe architectures.
5. Optimized Speculative Decoding: Integrated draft models accelerate inference by predicting multiple tokens per target forward pass. --suffix-decoding enables a model-free variant (SuffixDecoding, NeurIPS 2025) that drafts from the request's own prompt/output — no draft model, no extra VRAM, output guaranteed token-identical, with an automatic load-time probe that disables it on architectures whose caches can't roll back losslessly.
6. Optional Native Accelerators: opt-in C++/CUDA modules under native/ (a fused sampling kernel that collapses the per-token sampling pipeline into one launch, a grammar-bitmask kernel, and a C++ suffix cache) — build them once with uv pip install ./native/sampling ./native/suffix if you have MSVC + the CUDA toolkit, or don't: wLLM detects their absence and runs identically on the pure-torch paths.
7. Multi-Backend Choice: Native support for PyTorch, ONNX Runtime (via Optimum), and DirectML.
8. Production Observability: Prometheus /metrics endpoint with TTFT / latency / throughput percentiles.

The install.ps1 script (or install.bat wrapper) handles everything: Python 3.12 bootstrapping, virtual environment creation, CUDA-specific PyTorch wheels, and PATH configuration. No pre-installed Python required.

# Run the installer directly
powershell -ExecutionPolicy Bypass -File .\install.ps1

Environment (uv - Recommended):

uv venv .venv --python 3.12
.venv\Scripts\activate
uv pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
uv pip install -e . --extra-index-url https://download.pytorch.org/whl/cu124

Environment (standard pip):

python -m venv .venv
.venv\Scripts\activate
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
pip install -e . --extra-index-url https://download.pytorch.org/whl/cu124

Native accelerators (optional, needs MSVC Build Tools; native/sampling also needs the CUDA toolkit):

uv pip install ./native/suffix      # C++ suffix cache for --suffix-decoding
uv pip install ./native/sampling    # fused CUDA sampling + grammar bitmask kernels

Skip this entirely if you don't have the compilers — everything falls back to the pure-torch implementations automatically.

Interactive Chat:

winllm chat --model "microsoft/Phi-3-mini-4k-instruct" --quantization 4bit

API Server Deployment:

winllm serve --model "microsoft/Phi-3-mini-4k-instruct" --quantization 4bit --port 8000

Dive into the engineering principles behind the engine:

• SOLID Architecture — Package layout and the design principles behind it.
• Legacy Code Problems — What was wrong with the old flat layout and why it was restructured.
• Architecture Details — Visual guide to internal workflows.
• Genesys Deep Dive — From-first-principles inference guide.
• Walkthrough — End-to-end user guide.
• Changelog — Evolution of the engine.

MIT — wLLM is, and will always be, free and open source.