A local LLM inference engine written from scratch in Rust, with hand-written SIMD kernels and GPU compute shaders. No GGML, no llama.cpp — every matrix multiply, attention kernel, and quantization routine is implemented directly.

• Supported Models
• Downloading Models
• Features
• Binaries
• Design
• Benchmarks
• Building
• License

Note: Some Mistral models ship with tekken.json instead of tokenizer.json. The BF16 variants (recommended) include tokenizer.json directly.

Models are hosted on HuggingFace. Use the hf CLI to download:

# Install the HuggingFace CLI
pip install huggingface_hub

# Download a model (example: Qwen3-4B)
hf download Qwen/Qwen3-4B

# Download a vision-language model
hf download Qwen/Qwen3-VL-8B-Instruct

# Download a Mistral model (use BF16 variant for tokenizer.json)
hf download mistralai/Ministral-3-8B-Instruct-2512-BF16

Models are cached in ~/.cache/huggingface/hub/. Pass the snapshot path to --model:

herbert-cli --model ~/.cache/huggingface/hub/models--Qwen--Qwen3-4B/snapshots/<hash>

Mixed-precision supported: Q4, INT8, and BF16 weights in the same model.

• BF16 scalar — pure f32, no SIMD (reference/verification)
• BF16 AVX-512 — vdpbf16ps native BF16 dot-product (Zen4, Sapphire Rapids)
• INT8 AVX-512 — VPDPBUSD VNNI with fused QKV and gate+up projections
• Q4 AVX-512 — pre-interleaved nibble layout, huge pages, fused tile-local scales

All critical inner loops are hand-written x86-64 assembly (.S files).

• Metal (macOS) — 92 compute shaders covering Q4/INT8/BF16 matvec, matmul, flash attention, MoE, vision encoding, KV cache management
• Vulkan (Linux) — 31 GLSL compute shaders with cooperative matrix support, portable across AMD/NVIDIA/Intel. Use --gpu list to enumerate devices, --gpu N to select (0=first discrete, 1000+=global index for iGPU)

• Streaming token generation with UTF-8 multi-byte handling
• KV cache quantization (BF16, INT8)
• Thinking mode control for reasoning models (--nothink, --think-budget)
• Repetition loop detection
• Token sampling: temperature, top-k, top-p, greedy
• Mixture-of-Experts with batched expert dispatch
• Vision-Language support with multi-image input

Interactive chat or single-shot inference.

# Single-shot
herbert-cli --model <path> --prompt "What is 2+2?"

# Interactive chat
herbert-cli --model <path>

# With a system prompt
herbert-cli --model <path> --system "You are a helpful assistant."

# Vision (Qwen3-VL or Pixtral models)
herbert-cli --model <path> --image photo.jpg --prompt "Describe this image."

# Greedy decoding (temperature=0)
herbert-cli --model <path> --temperature 0 --prompt "Hello"

# Show stats after generation
herbert-cli --model <path> --prompt "Hello" --verbose

# Choose a specific backend
herbert-cli --model <path> --backend metal-q4
herbert-cli --model <path> --backend help   # list available backends

# GPU selection (Vulkan)
herbert-cli --gpu list                       # list available GPUs
herbert-cli --gpu 0 --model <path> --backend vulkan-bf16   # first discrete GPU (default)
herbert-cli --gpu 1000 --model <path> --backend vulkan-bf16 # iGPU (global index)

# Tool calling
herbert-cli --model <path> --tools

Sampling options: --temperature (default 0.4), --top-k (default 40), --top-p (default 0.9), --max-tokens (default 2048)

Chat commands: /help, /config, /temp, /topk, /topp, /think, /nothink, /tools, /image, /stats, /arch, /clear, /quit

Built-in tools (with --tools): get_datetime, calculate, list_directory, read_file

Anthropic Messages API compatible server with SSE streaming.

herbert-server --model <path> --addr 0.0.0.0:3000
herbert-server --model <path> --addr 0.0.0.0:3000 --api-key mysecretkey

Endpoints:

• POST /v1/messages — chat completion (streaming SSE or JSON)
• POST /v1/messages/count_tokens — token counting
• POST /v1/tokenize — tokenization
• GET /v1/metrics — performance metrics

Native desktop application built with Tauri 2 and Vue 3.

• Multi-conversation chat with sidebar
• Streaming with markdown rendering and syntax highlighting
• Image support with drag & drop and encoding progress
• Per-message performance stats
• Model loading with progress feedback
• Settings panel for sampling parameters and backend selection

Most inference engines optimize for prefill throughput (batched GEMM). Herbert takes a different approach: it is built around the assumption that inference performance is limited by memory bandwidth, not compute. It prioritizes decode speed, which is what determines the user experience in interactive use.

At decode time, the bottleneck is memory bandwidth — each generated token requires reading the full KV cache. Herbert addresses this with an INT8 KV cache that halves the bandwidth requirement compared to FP16, using hand-written VNNI kernels that avoid the dequantization overhead seen in other implementations.

In practice:

• Prefill: not the primary optimization target yet (batched matmul is in progress). Currently 1.5-2x behind llama.cpp on dense models
• Decode (short context): on par with llama.cpp
• Decode (long context): performance improves as context grows, because KV cache bandwidth becomes the dominant cost — and Herbert reduces that cost

On Mixture-of-Experts models, Herbert's expert batching also improves prefill, leading to better performance across the board.

These results are consistent across all four tested models and architectures (dense, VL, MoE).

Every kernel optimization is validated empirically using:

• Hardware performance counters (AMD Zen4 PMC via perf_event_open + rdpmc fast-path) — cycle-precise, core-pinned, multi-pass measurement of L1/L2/L3 cache behavior, retired instructions, and branch mispredictions
• Memory bandwidth sweeps — working set sizes from L1 (48KB) through L2 (1.25MB) to DRAM (64MB+) to establish theoretical bandwidth ceilings
• Wall-clock throughput — end-to-end prefill and decode measurements with controlled cooldown periods between runs

CPU-only benchmarks on an AMD Ryzen 9 7900 (12C/24T, AVX-512, 96 GB DDR5), comparing Herbert with llama.cpp, HF Transformers, vLLM-CPU, and ONNX Runtime.

Herbert's KV cache quantization (INT8 by default) gives it an increasing advantage over llama.cpp as context length grows. On short contexts, llama.cpp is slightly faster; on longer contexts (1K+ tokens), Herbert pulls ahead.

Qwen3-0.6B — Q4 decode

Qwen3-VL-30B-A3B (MoE) — Q4 decode

On MoE models, Herbert also wins on prefill thanks to its expert batching optimizations (moe-v6).

llama.cpp has faster prefill on dense models (~1.5-2x) due to its batched GEMM. vLLM-CPU has the best prefill overall thanks to chunked prefill + torch matmul.

Qwen3-0.6B — prefill at ~1000 tokens

Full benchmarks with all prompt sizes, quantizations, and KV cache configurations are in docs/benchmarks/:

• Qwen3-0.6B — dense 0.6B, 5 engines including ONNX Runtime
• Qwen3-VL-4B — dense VL 4B
• Ministral-3 3B — dense text 3B (Mistral)
• Qwen3-VL-30B-A3B — MoE 30B (3B active)

cargo build --release

The build auto-detects available CPU features (AVX-512, VNNI, AVX-512 BF16) and compiles the appropriate assembly kernels. Metal shaders are compiled on macOS, Vulkan shaders on Linux.

MIT — see LICENSE