Pure-Rust LLM engine: serving, agents, and on-policy distillation — on Apple Silicon and NVIDIA. No Python on the hot path.

_{35B MoE at 85 tok/s on a MacBook · bit-identical speculative decode · OPD lifts 4B student +27pp on MATH-500}

Quick Start · Performance · Why ARLE · HTTP API · Support Matrix · Architecture · Changelog

English · 简体中文

# Apple Silicon (Homebrew)
brew install cklxx/tap/arle

# Apple Silicon or Linux x86_64 (one-line installer)
curl -fsSL https://github.com/cklxx/arle/releases/latest/download/install.sh | sh

# Linux + NVIDIA (Docker, no compile needed)
docker run --rm --gpus all -p 8000:8000 -v $PWD/models:/models:ro \
 ghcr.io/cklxx/arle:latest serve --backend cuda --model-path /models/Qwen3.5-4B

# NVIDIA CUDA
arle serve --backend cuda --model-path /path/to/Qwen3.5-4B --port 8000

# Apple Silicon (Metal)
arle serve --backend metal --model-path mlx-community/Qwen3.6-35B-A3B-4bit --port 8000

# DSpark block-drafter decode + in-process Markov-head training (CUDA)
arle serve --backend cuda \
 --model-path /path/to/Qwen3.6-27B \
 --spec-type dspark --mtp-draft-model /path/to/dspark-draft \
 --dspark-train --port 8000

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
print(client.chat.completions.create(
 model="default",
 messages=[{"role": "user", "content": "Hello from ARLE"}],
).choices[0].message.content)

Source builds need a backend. cargo build --release alone produces a CLI-only binary. Add --features cuda (NVIDIA) or --no-default-features --features metal,no-cuda,cli (Apple Silicon). See docs/install.md.

Full install matrix, uninstall, and build from source: docs/install.md · Examples: examples/.

Measured on real hardware, not projected.

A 35B-A3B MoE decodes as fast as the 4B dense — only ~3B params activate per token.

Qwen3.6-27B (OptiQ 4/8-bit): the model's own NextN/MTP head drafts, the base verifies — output is bit-identical to greedy, 12.3 → 17.75 tok/s (+44%), past the 15.2 tok/s HBM floor no kernel can reach.

Measured on H20 (96 GB), --spec-type mtp --mtp-draft-tokens 2, greedy verification (topk=1).

The 35B-A3B hybrid MoE (30 linear-attention + 10 full-attention layers, 256 experts, 8/tok) decodes ~2× faster than the 27B dense model at the same quant level. MTP gains are modest at greedy topk=1; higher draft token counts and topk can increase the speedup.

Teacher = production server. Student trains on its own rollouts:

• Qwen3.5-4B: MATH-500 +27pp (0.518 → 0.792)
• Qwen3.5-27B: Terminal-Bench pass@1 +5.1pp (20.5 → 25.6%)

Method and raw data: benchmarks/README.md · docs/experience/wins/.

Agent and RL workloads re-process the same prompt + history + tool output every turn. ARLE fixes this once and shares the fix across serving and training.

KV stays hot across turns. Prior-turn KV stays on GPU; prefix pages are shared across requests via the host radix cache, demote to host RAM under memory pressure, and promote back on next hit — no re-prefill.

Quantized KV on CUDA. INT8/FP8/INT4 paged-KV behind --kv-cache-dtype. Correctness-gated, opt-in (default BF16).

KV-recall for long context (Metal). Past the sliding window, decode attends only sink + recent + top-k recalled older blocks — 9.6% of KV, identical quality to full attention. Behind --kv-recall.

One runtime, three surfaces. Serving, the local agent, and OPD training run the same Rust + model code. The OPD teacher is the production server.

DSpark trains while serving. --spec-type dspark --dspark-train runs the DSpark block-drafter for faster decode and trains its Markov head in-process: the hot path captures (draft, target, accepted) tuples, a background thread runs acceptance-weighted policy gradient + probability matching, and updated weights hot-swap back into the running engine — no restart, no separate training job. Seeds from the loaded checkpoint so acceptance never regresses at startup.

flowchart TB
 classDef entry fill:#1a1a2e,stroke:#e94560,color:#eee,rx:8,ry:8
 classDef core fill:#0f3460,stroke:#e94560,color:#eee,rx:8,ry:8
 classDef seam fill:#533483,stroke:#e94560,color:#eee,rx:8,ry:8
 classDef exec fill:#190e36,stroke:#4ecca3,color:#eee,rx:8,ry:8

 Serve["arle serve<br/><sub>OpenAI v1</sub>"]
 Agent["arle<br/><sub>local agent</sub>"]
 Train["arle train opd<br/><sub>OPD</sub>"]

 Core["infer-core<br/><sub>device-neutral Engine · scheduler · KV cache</sub>"]

 Seam["infer-seam<br/><sub>two traits: BackendExecutor · KvPool</sub>"]

 CUDA["infer-cuda<br/><sub>FlashMLA · DeepGEMM · DeepEP</sub>"]
 Metal["infer-metal<br/><sub>MLX bridge</sub>"]

 Serve --> Core
 Agent --> Core
 Train --> Core
 Core --> Seam
 Seam --> CUDA
 Seam --> Metal

 class Serve,Agent,Train entry
 class Core core
 class Seam seam
 class CUDA,Metal exec

One runtime, three surfaces, two pluggable backends. A new backend implements the two seam traits — no changes to the scheduler, cache, or server.

Deep dive: docs/onboarding.md (30 min) · docs/architecture.md · docs/codebase-map.md.

Full tiers: docs/support-matrix.md · docs/stability-policy.md.

HTTP API · Support Matrix · Architecture · Codebase Map · Environment · Troubleshooting · Contributing · All docs

MIT