Tags: y0sif/ramvamp

ramvamp v0.1.0

First public release. Runs Qwen3-30B-A3B in about 3 GB of RAM on CPU by
streaming experts from NVMe.

Measured cold on the reference machine, a DRAM-less QLC drive, inside a 3 GB
cgroup with swap off: decode 1.46 to 2.19 tok/s across ctx 64 to 3,961,
prefill 11.25 tok/s at ctx 512, peak 2,497 to 2,929 MiB of a 3,072 MiB
ceiling. Fidelity against llama.cpp on byte-identical weights: mean full-vocab
KL 1.04e-2, top-1 agreement 8 of 8.

Throughput is stated per drive on purpose. Decode is I/O-bound and this part
sustains about 1.6 GB/s at decode's own read geometry.