Pure-Rust LLM engine: serving, agents, and on-policy distillation — on Apple Silicon and NVIDIA. No Python on the hot path.

_{35B MoE at 85 tok/s on a MacBook · bit-identical speculative decode · OPD lifts 4B student +27pp on MATH-500}

Quick Start · Performance · Why ARLE · HTTP API · Support Matrix · Architecture · Changelog

English · 简体中文

# Apple Silicon (Homebrew)
brew install cklxx/tap/arle

# Apple Silicon or Linux x86_64 (one-line installer)
curl -fsSL https://github.com/cklxx/arle/releases/latest/download/install.sh | sh

# Linux + NVIDIA (Docker, no compile needed)
docker run --rm --gpus all -p 8000:8000 -v $PWD/models:/models:ro \
 ghcr.io/cklxx/arle:latest serve --backend cuda --model-path /models/Qwen3.5-4B

# NVIDIA CUDA
arle serve --backend cuda --model-path /path/to/Qwen3.5-4B --port 8000

# Apple Silicon (Metal)
arle serve --backend metal --model-path mlx-community/Qwen3.6-35B-A3B-4bit --port 8000

# DSpark block-drafter decode + in-process Markov-head training (CUDA)
arle serve --backend cuda \
 --model-path /path/to/Qwen3.6-27B \
 --spec-type dspark --mtp-draft-model /path/to/dspark-draft \
 --dspark-train --port 8000

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
print(client.chat.completions.create(
 model="default",
 messages=[{"role": "user", "content": "Hello from ARLE"}],
).choices[0].message.content)

Source builds need a backend. cargo build --release alone produces a CLI-only binary. Add --features cuda (NVIDIA) or --no-default-features --features metal,no-cuda,cli (Apple Silicon). See docs/install.md.

Full install matrix, uninstall, and build from source: docs/install.md · Examples: examples/.

Measured on real hardware, not projected.

A 35B-A3B MoE decodes as fast as the 4B dense — only ~3B params activate per token.

Qwen3.6-27B (OptiQ 4/8-bit): the model's own NextN/MTP head drafts, the base verifies — output is bit-identical to greedy, 12.3 → 17.75 tok/s (+44%), past the 15.2 tok/s HBM floor no kernel can reach.

Measured on H20 (96 GB), --spec-type mtp --mtp-draft-tokens 2, greedy verification (topk=1).

The 35B-A3B hybrid MoE (30 linear-attention + 10 full-attention layers, 256 experts, 8/tok) decodes ~2× faster than the 27B dense model at the same quant level. MTP gains are modest at greedy topk=1; higher draft token counts and topk can increase the speedup.

Teacher = production server. Student trains on its own rollouts:

• Qwen3.5-4B: MATH-500 +27pp (0.518 → 0.792)
• Qwen3.5-27B: Terminal-Bench pass@1 +5.1pp (20.5 → 25.6%)

Method and raw data: benchmarks/README.md · docs/experience/wins/.

Agent and RL workloads re-process the same prompt + history + tool output every turn. ARLE fixes this once and shares the fix across serving and training.

KV stays hot across turns. Prior-turn KV stays on GPU; prefix pages are shared across requests via the host radix cache, demote to host RAM under memory pressure, and promote back on next hit — no re-prefill.

Quantized KV on CUDA. INT8/FP8/INT4 paged-KV behind --kv-cache-dtype. Correctness-gated, opt-in (default BF16).

KV-recall for long context (Metal). Past the sliding window, decode attends only sink + recent + top-k recalled older blocks — 9.6% of KV, identical quality to full attention. Behind --kv-recall.

One runtime, three surfaces. Serving, the local agent, and OPD training run the same Rust + model code. The OPD teacher is the production server.

DSpark trains while serving. --spec-type dspark --dspark-train runs the DSpark block-drafter for faster decode and trains its Markov head in-process: the hot path captures (draft, target, accepted) tuples, a background thread runs acceptance-weighted policy gradient + probability matching, and updated weights hot-swap back into the running engine — no restart, no separate training job. Seeds from the loaded checkpoint so acceptance never regresses at startup.

flowchart TD
  CLI["CLI orchestration<br/>crates/cli/src/train_cli.rs"]
  OPD["OPD / self-OPD<br/>student rollout + teacher rescoring<br/>KL / reverse-KL / beta-JSD"]
  RUBRIC["Rubric-OPD<br/>sample -> judge -> accepted CE"]
  AGENT["Agent-OPD<br/>tool trajectory -> reward<br/>UpdatePreset"]
  DSPARK["DSpark online training<br/>verify-logit capture -> Markov-head update"]
  STUDENT["Qwen3.5 / Qwen3.6 train model<br/>crates/train/src/qwen35.rs"]
  TEACHER["infer-api or EMA teacher<br/>BF16 logits boundary"]
  LOSS["Loss graph<br/>CE / KL / JSD / weighted PG"]
  TAPE["Autograd Tape<br/>checkpoint / recompute / offload"]
  BACKEND["Backend seam<br/>CPU reference + optional CUDA overrides"]
  OPT["Production host AdamW<br/>FP32 params and moments"]
  SYNC["LoRA D2H<br/>infer-engine re-merge"]
  ART["Immutable model/adapter artifact<br/>publish latest last"]
  CODEC["Trainer-state v2 codec<br/>not wired to production CLI"]

  CLI --> OPD
  CLI --> RUBRIC
  CLI --> AGENT
  CLI --> DSPARK
  OPD --> STUDENT
  OPD --> TEACHER
  RUBRIC --> STUDENT
  AGENT --> STUDENT
  STUDENT --> LOSS
  TEACHER --> LOSS
  LOSS --> TAPE
  TAPE --> BACKEND
  BACKEND --> OPT
  OPT --> SYNC
  OPT --> ART
  CODEC -. "no production call edge" .-> ART
  DSPARK -->|"two BF16 vocab-wide D2H copies"| DSPCPU["CpuBackend + host AdamW"]
  DSPCPU -->|"BF16 saved head / host hot-swap"| SYNC

The training stack shares the production model/runtime authority: serving or EMA teachers score student trajectories, one autograd substrate drives OPD and RFT objectives, and LoRA updates merge back into the live inference engine. The current device-residency and restart gaps are documented in the training architecture and kernel audit.

Deep dive: docs/onboarding.md (30 min) · docs/architecture.md · docs/codebase-map.md.

Full tiers: docs/support-matrix.md · docs/stability-policy.md.

HTTP API · Support Matrix · Architecture · Codebase Map · Environment · Troubleshooting · Contributing · All docs

MIT