ds4-rs

A Rust port of antirez/ds4: a single-model inference engine for DeepSeek V4 Flash.

Status: Phase 1 forward pass is numerically complete; the DS4_TEST_MODEL smoke run now completes end-to-end against a real DS4 GGUF (prefill plus decode steps) without panicking. Full numerical validation still needs a healthy GGUF — the q2-imatrix DS4-Flash file in hand has corrupted F16 weights and produces the same gibberish on the antirez/ds4 C reference. Interactive REPL and session lifecycle (rewind, invalidate) are wired up. See todo.md for the current backlog.

• Load DeepSeek V4 Flash GGUF weights and run greedy token generation on Linux.
• Keep the architecture simple, modular, and easy to hack on.
• Defer GPU backends (Metal, CUDA, Vulkan) to future work.

cargo build --release

# One-shot generation (use -p)
ds4 -p "What is Rust?" -n 128

# Interactive REPL (default when no -p is supplied)
ds4

# With explicit model path and context size
ds4 --model ./ds4flash.gguf -p "Hello" -n 64 --ctx 8192

In the REPL, /help (or /?) lists the slash commands: /reset (/clear), /rewind <n>, /ctx, /exit (/quit).

cargo test --workspace

• GGUF v3 — memory-mapped for zero-copy weight access.
• BPE tokenizer — loaded from GGUF metadata.
• Quantization — mixed-dtype kernels (Q8_K activations, IQ2_XXS / Q2_K / Q4_K / IQ4_XS / IQ4_NL weights) with matmul dispatch.
• MLA latent KV cache — caches compressed key/value state.
• MoE routing — shared expert + routed experts (top-k gating with hash routing for early layers).

See PLAN.md and rfcs/ for design details.

• Linux
• Rust toolchain (see rust-toolchain)

MIT — see LICENSE.