ramvamp

Run 26-30B MoE models in about 3 GB of RAM. No GPU required.
A Rust runtime that streams experts from NVMe instead of holding them hostage in memory.

Status: v0, and it runs. Qwen3-30B-A3B generates coherent text on CPU inside a 3 GB cgroup with a cold page cache, validated against llama.cpp on identical weights. The shipping surface is ramvamp serve: the OpenAI Chat Completions API on loopback, with streaming SSE, tool calls, and named configuration profiles.

ramvamp is a CPU-first local LLM inference runtime, written in Rust, for fine-grained Mixture-of-Experts models. It runs a 30B-parameter quantized model on an ordinary Linux x86_64 machine with an NVMe SSD, no GPU and no 32 GB of RAM, by never loading the full checkpoint into memory. The always-needed common weights stay memory-mapped; the routed experts, which are most of the model, live on disk in a page-aligned packed format and are fetched with io_uring and O_DIRECT only when the router asks for them, through a small per-layer LFU cache. Qwen3-30B-A3B is the v0 model; Gemma 4 26B-A4B is the intended second model and is roadmap, not present tense, since the repacker accepts qwen3moe architecture GGUFs and refuses anything else by name.

Qwen3-30B-A3B Q4_K_M, cold, inside memory.max=3G with memory.swap.max=0, on the reference machine (Intel Core Ultra 9 185H, CPU only, Micron 2400 DRAM-less QLC NVMe):

Throughput is stated per drive on purpose: decode is I/O-bound and this part sustains about 1.6 GB/s at decode's own read geometry, so no figure appears without its drive. Full table, conditions and drive detail: docs/benchmarks.md.

curl -sSL https://y0sif.github.io/ramvamp/install.sh | bash

Puts ramvamp and ramvamp-repack in /usr/local/bin. Pin a tag with RAMVAMP_VERSION=v0.1.0, move the destination with RAMVAMP_INSTALL_DIR, re-run the same line to upgrade. Linux x86_64 only, and on anything else it refuses and installs nothing: io_uring is a Linux interface, the fast kernels are AVX2 plus F16C, and a binary elsewhere would run on fallback paths that no number here describes.

Then install the model. That is a 17.35 GiB download, so the installer prints this and stops rather than starting it for you, and the path wants to be on an NVMe SSD:

# Download and repack the pinned Qwen3-30B-A3B in one streaming pass.
ramvamp-repack install --output ~/models/qwen3-30b-a3b.rvmp

# Generate. Text streams to stdout; timing and expert stats go to stderr.
ramvamp generate --model ~/models/qwen3-30b-a3b.rvmp \
    --prompt "Explain io_uring in two sentences."

curl -sSL -o ramvamp.tar.gz \
  https://github.com/y0sif/ramvamp/releases/latest/download/ramvamp-linux-x86_64.tar.gz
tar xzf ramvamp.tar.gz
sudo install -m755 ramvamp ramvamp-repack /usr/local/bin/

git clone https://github.com/y0sif/ramvamp
cd ramvamp
cargo build --release

Requirements, the three I/O modes and the repacker's other subcommands: docs/install.md.

The quickest way to try it, with nothing to wire up:

ramvamp chat --model ~/models/qwen3-30b-a3b.rvmp

# Same thing with a live status panel: prefill progress, tokens per second,
# context used, and the expert cache hit rate, updated while it generates.
ramvamp chat --model ~/models/qwen3-30b-a3b.rvmp --tui

--tui exists because a long prompt is minutes of prefill before the first character appears, and silence is indistinguishable from a hang. It needs a terminal on stdout. Plain chat is the reference behaviour and stays that way.

ramvamp serve --model ~/models/qwen3-30b-a3b.rvmp --port 8080

That is /v1/chat/completions (streaming SSE and buffered), /v1/models and /health, on 127.0.0.1 with no --host flag, on purpose. It serves one request at a time, reuses the KV cache across requests by longest common prefix, and refuses fields it cannot honour exactly as typed 400s rather than accepting them quietly. Routes, refusals and positioning: docs/server.md.

--context (default 4096) is the dial that trades conversation length against resident bytes, and plan prices any value of it without loading the model; a 32K profile projects about 5,649 MiB and needs a bigger budget than 3 GB, which is what named profiles exist to express. Profiles, --profile / --config / --no-config, the four-layer precedence chain and the three environment variables: docs/configuration.md.

1. Install repacks, it does not convert. The GGUF is streamed into the .rvmp layout: a common.bin for what every token needs, one file per layer holding 128 fixed-stride, page-aligned expert blobs. Nothing is requantized and no shard is materialized in heap memory.
2. Common weights are mmap'd. Embeddings, attention, routers, norms and the lm_head are about 1,023 MiB, touched every token, and the page cache keeps them resident.
3. Experts are read explicitly, never demand-paged. Per layer the router picks 8 of 128. Hits in that layer's LFU cache dispatch to the compute pool immediately; the misses go to io_uring as one batch of O_DIRECT reads, so they bypass the page cache and cost the memory budget nothing, and the hit compute hides their latency.
4. The cache is small on purpose. At the default 1,440 MiB budget that is 11 slots per layer out of 128 experts. Worst case is 1,097 MB of expert weights per decode token, and the cache absorbs 53.0% to 59.3% of requests across five measured context rungs (EXP-023).

Explicit reads rather than mmap demand paging is a measured choice, and so is having no speculative cross-layer expert prefetch; docs/landscape.md carries both measurements. docs/architecture.md is the design and docs/roadmap.md the plan of record.

cargo build
cargo test
cargo clippy --all-targets -- -D warnings
cargo fmt --check

All four must pass before a push. CONTRIBUTING.md covers the rest, including the measurement rules a performance change has to follow; security issues go through SECURITY.md.

Inspired by TurboFieldfare, a Swift + Metal runtime that proved the physics on Apple Silicon: Gemma 4 26B-A4B in about 2 GB at 5.1 to 6.3 tok/s on an 8 GB M2 Air. ramvamp exists because that proof is locked to Macs with M-series chips.

Licensed under either of Apache License 2.0 or MIT License at your option. Model weights are not included and remain governed by their own terms.