DeepSeek-V4.1-Flash-1017 (552 B + 196 B Engram) on one used 8x V100 server
2,400-3,000 tokens/s across hundreds of sessions · DSpark speculative decoding · OpenAI- and Anthropic-compatible
API · thinking levels, streaming, prefix reuse
An 8x V100 machine (the DGX-1 NVLink layout, 256 GB of HBM) is old and cheap, and no serving engine runs DeepSeek-V4.1-Flash-1017 (DeepSeek, MIT license) on it: Volta has no FP8, FP4 or bf16 math, and the official checkpoint is larger than its memory. Together the eight cards still read 6.9 TB/s. volta is an engine in Rust and CUDA, written for this one model on this one class of machine, that turns that bandwidth into throughput for many sessions at once.
- Fast: 2,400 tokens/s across 256 sessions and 3,070 across 512; 24 tokens/s per session with 64 at once. A
single session decodes 53 tokens/s, and 76 with
--head-parallel(~230 on code with the draft). - Close to the original: the same next token as the FP8 model 84% of the time on the Lucebox eval (KL 0.127).
- Faster with the draft model: the checkpoint's DSpark draft runs on the GPUs. With 64 sessions writing code, each gets 27 tokens/s instead of 20, and sampled outputs keep the model's own distribution (speculative sampling).
- Plugs into your tools: the OpenAI- and Anthropic-compatible API of dsx, with streaming, tools, thinking levels and the chat in the browser.
- Long context: 128K tokens per session for 16 sessions at once (256K with one per GPU). The compressed KV cache is stored as FP4, ~0.9 KB per token.
- Picks up where you left off: a finished session's KV cache stays on the GPU, so an agent that resends the whole conversation only pays for the new text.
https://huggingface.co/lxe/DeepSeek-V4.1-Flash-1017-Volta
The Hugging Face repository contains one complete DeepSeek-V4.1-Flash-1017-Volta.gguf model file, about
442 GB. It includes the unchanged base tensors, Engram tables, draft and vision weights, tokenizer and model
configuration, together with the 220 GB of quantized main routed experts. No separate base directory or tokenizer
file is required. The experts use cb3 (15,172 experts, 3.25 bits per weight) and cb2 (188 cold experts, 2.25 bits),
chosen by how often the router picks them.
The GGUF format stores row-major expert codes and scale bytes. The engine reconstructs its kernel layout at load while preserving the quantized values and original eight-way source slices. The file uses custom Volta encoding metadata and requires this engine. It is not a llama.cpp quant type. Previously downloaded native expert packs and base directories remain supported.
The complete GGUF also has GPU validation and four-agent measurements from October 24, 2026, including actual Pi tool use, live tmux shapes, 128K/256K retrieval checks and controlled sampler comparisons. The data-parallel 128K profile measured 165.2–169.3 aggregate whole-request tokens/s for four cold 4K-input/1,024-output requests. It remains the measured throughput choice for four short-task agents.
For head-parallel serving, VOLTA_HP_GPU_SAMPLE=1 opts into collecting the eight vocabulary slices on GPU0 and using the existing GPU sampler. Four-client, 256K-capacity A/B/B/A tests measured 51.8–53.9% higher fixed-workload throughput and 47.8–47.9% higher mixed prefill/decode throughput than CPU HP sampling. The HP receipts retain the real-row audit, quality checks, fallbacks and different CPU/GPU sampling trajectories. Default HP sampling is unchanged.
The quality and performance results below were measured with the original native pack, before the GGUF repack.
Against the FP8 model on the Lucebox eval (8 x 1,024 tokens), it picks the same next token 84.1% of the time, with KL 0.127 and perplexity 8.88.
All on one 8x V100-SXM2-32GB machine. Engine benchmark (volta bench, real text, one token per session per step):
| Sessions | Step | Total |
|---|---|---|
| 64 | 41.7 ms | 1,536 tok/s |
| 128 | 69 ms | 1,850 tok/s |
| 256 | 105 ms | 2,440 tok/s |
| 512 | 167 ms | 3,070 tok/s |
With the DSpark draft (volta spec --adaptive, greedy, 64 sessions): 28.4 tok/s per session on code and 18.2 on
prose. With 8 sessions on code: 81 tok/s each.
One session (volta spec, greedy, steady-state step):
| Mode | Step | Per session | With the draft, on code |
|---|---|---|---|
| default (attention on the session's GPU) | 17.8 ms | 56 tok/s | 200 tok/s |
--head-parallel (every GPU, 8 heads each) |
10.7 ms | 94 tok/s | 277 tok/s |
With --head-parallel, 4 sessions with the draft on code get 114 tok/s each.
Through the server (volta-server, sampling at temperature 1 with drafts, 300-token answers; up to 8 streams with
--slots 1 --ctx 131072, 32 with --slots 4 --ctx 32768):
| Concurrent streams | Per stream | Total | First token |
|---|---|---|---|
| 1 | 93 tok/s | 84 tok/s | 0.33 s |
| 8 | 72 tok/s | 343 tok/s | 0.5 s |
| 32 | 39 tok/s | 967 tok/s | 1.0 s |
Long prompts (volta spec, 8 sessions, one per GPU, code, with drafts): with 64K-token prompts, prefill runs at
2,490 tok/s in total and decoding at 71 tok/s per session; with 128K-token prompts, at 1,430 and 46. The compressed
KV cache is stored as FP4, ~0.9 KB per token per session, so --slots 2 --ctx 131072 gives 16 sessions of 128K.
For comparison, llama.cpp (PR #30817, UD-IQ1_M) on the same machine decodes 26.7 tok/s for one stream and 72-97 tok/s in total at most.
Download the complete model (~442 GB):
hf download lxe/DeepSeek-V4.1-Flash-1017-Volta --local-dir volta-weightsBuild and start the server:
NVCC=/usr/local/cuda-12.9/bin/nvcc CUDA_ARCHS=70 cargo build --release -p dsx --features cuda --bin volta-server
./target/release/volta-server \
--model volta-weights/DeepSeek-V4.1-Flash-1017-Volta.gguf \
--host 0.0.0.0 --port 8090That serves 32 sessions of 8,192 tokens. For agents with long conversations, trade sessions for context:
--slots 2 --ctx 131072 serves 16 sessions of 128K, and --slots 1 --ctx 262144 8 sessions of 256K.
For the fastest single sessions, add --head-parallel: every GPU runs every session with 8 of the 64 attention
heads, so the machine holds --slots sessions in total instead of --slots per GPU, each at up to ~94 tok/s
without the draft. Every GPU keeps a copy of each session's KV cache, so --slots and --ctx share memory as
they do without the flag.
Loading reconstructs the expert kernel layout and maps the 196 GB Engram tables into RAM. Full-model startup
time for the GGUF path has not been measured. Open http://localhost:8090/
for the web UI, or point any OpenAI client at http://localhost:8090/v1 (Anthropic clients at
http://localhost:8090):
curl http://localhost:8090/v1/chat/completions \
-H 'content-type: application/json' \
-d '{
"model": "deepseek-v4.1-flash",
"stream": true,
"reasoning_effort": "none",
"messages": [
{"role": "user", "content": "Write a haiku about MoE routing."}
]
}'An admin dashboard runs on http://localhost:8091/: output and prompt throughput (live and over the last 5 minutes),
sessions in use, speed per session, drafts kept, step time, and the active and recently finished requests. The same
numbers are JSON at /stats.
To use the official checkpoint instead of the complete GGUF, pass the checkpoint directory as
--model: the engine packs the experts on the GPUs at load (~3 minutes) and caches them in /dev/shm/vcache.
For the pi coding agent, add this to ~/.pi/agent/models.json:
{
"providers": {
"volta": {
"baseUrl": "http://localhost:8090/v1",
"api": "openai-completions",
"apiKey": "none",
"authHeader": false,
"compat": {
"supportsDeveloperRole": false,
"supportsReasoningEffort": true,
"maxTokensField": "max_tokens"
},
"models": [
{
"id": "deepseek-v4.1-flash",
"reasoning": true,
"input": ["text"],
"contextWindow": 131072,
"maxTokens": 16384,
"thinkingLevelMap": { "off": "none", "high": "high" }
}
]
}
}
}pi --provider volta --model deepseek-v4.1-flash:offcontextWindow must match the server's --ctx (here --slots 2 --ctx 131072 --max-tokens 16384).
pi compacts long conversations by itself (it summarizes older messages when the context is nearly full). To start
compaction with room for a full answer and keep more recent history, add this to ~/.pi/agent/settings.json:
{
"compaction": {
"enabled": true,
"modelOverrides": {
"volta/deepseek-v4.1-flash": { "reserveTokens": 20480, "keepRecentTokens": 32768 }
}
}
}The server's prefix cache reuses everything before the summary, so the first request after a compaction only reads the summary and the messages kept after it.
With --api-key: drop "authHeader": false, set "apiKey" to the key and point baseUrl at the server's URL.
| GPUs | 8x V100 32 GB with NVLink in the DGX-1 layout (SXM2 or SXM3) |
| CUDA | 12.9 (the last toolkit that builds for sm_70) |
| RAM | ~400 GB (the Engram tables, 196 GB, are read from RAM) |
| Disk | ~442 GB for the Hugging Face build; use fast storage for the model file |
Other machines need other kernels and collectives; DESIGN.md lists what each would take.
volta-server needs --model; every other option has a default.
Model files
--model |
the complete Volta GGUF, or a checkpoint/base directory |
VOLTA_PACK |
optional expert override for directory models: GGUF or native pack (original GPU topology required) |
--tokenizer |
tokenizer.json; default: the model directory's |
Server
--host, --port |
listen address (default 127.0.0.1:8090) |
--api-key |
require this key (Authorization: Bearer <key> or x-api-key; env DSX_API_KEY) |
--model-name |
model id reported by /v1/models (default deepseek-v4.1-flash) |
--slots |
concurrent sessions per GPU (default 4); with --head-parallel, in total |
--head-parallel |
every GPU runs every session with 8 of the 64 attention heads and a copy of its KV cache: the fastest single sessions, for fewer of them |
--ctx |
context window per session, prompt + generated tokens (default 8192); slots and context share the GPUs' spare memory: 2 x 131072 fits |
--max-tokens |
generation cap when a request sets none (default 4096) |
--chunk |
prompt tokens per session per step while reading a prompt (default 32) |
--admin-port |
the admin dashboard and /stats (default 8091, on --host; 0 = off). It has no API key: keep it on loopback or a private network |
Sampling defaults (a request's own fields override them)
--temperature, --top-p |
default 1.0 and 1.0, the model's recommended sampling |
--hesitation-penalty |
logit penalty on hesitation tokens ("Wait", "But", ...) while reasoning; 0 = off (default) |
Draft
--spec-k |
most draft tokens verified per pass (default 5; 0 turns the draft off) |
--theta |
a session verifies j drafts while p^j >= theta, p = its recent per-draft acceptance (default 0.35) |
Thinking is on by default, at high effort. A request turns it off or sets the effort:
OpenAI API (/v1/chat/completions) |
Anthropic API (/v1/messages) |
|
|---|---|---|
| Thinking off | "reasoning_effort": "none" or "thinking": {"type": "disabled"} |
"thinking": {"type": "disabled"} |
| Effort | "reasoning_effort": low (or minimal), high (or medium), xhigh, max |
"output_config": {"effort": ...}: low, high (or medium), xhigh, max |
The model has 15,360 routed experts (384 in each of its 40 layers); each token uses 6 per layer, plus a shared expert, attention, and two Engram lookups. Every step runs all sessions' tokens together.
- Every expert split 8 ways: each GPU holds 1/8 of every expert's weights (and of the shared expert), so every GPU does the same work in every layer, whichever experts the router picks.
- Formats made for the hardware: each expert's MXFP4 weights become cb3 or cb2 codes. The kernel turns two
weights into fp16 with one shared-memory lookup and feeds Volta's
mma.m8n8k4tensor cores directly. The dense weights are 4-bit with an fp16 scale per 16. - Attention per GPU, experts across all of them: each GPU keeps its own sessions' KV cache and runs their attention and dense layers. Per layer, one NVLink all-gather sends every token's hidden state to every GPU, and one all-to-all brings the partial sums home. Both are single kernels that write into peer memory and signal with flags (~13-17 µs each).
- Head-parallel for few sessions: with
--head-paralleleach GPU runs 8 of the 64 heads (one of the model's 8 output groups) of every session, so a token's dense attention weights are read once across the GPUs instead of whole on one. The attention output and the MoE partials are summed on every GPU in a fixed order, so all eight hold the same bits and route to the same experts; small exchanges use NCCL-style LL words (data and sequence number in one 8-byte store), 5-8 µs. - Attention on the tensor cores: one KV row serves all 64 query heads, so a token's attention is a matrix product,
run with
mma.m8n8k4in fp16. The indexer that picks each token's 512 compressed rows scores them on the tensor cores too. - KV cache in FP4: the model rounds its compressed KV rows and indexer keys to FP4, so they are stored packed, exactly: ~0.9 KB per token per session, 0.23 GB for a 256K-token context.
- One CUDA graph per step: routing is built on the GPU, small kernels are merged, and independent work (the shared expert, the larger expert batches) runs on a second stream inside the graph.
- Several tokens per session per step: prompts are read in chunks, and the draft's tokens are verified, in the same step as everyone else's decoding.
- DSpark on the GPUs: the checkpoint's 3-layer draft runs through the same MoE path. Each session verifies as many drafts as its recent acceptance justifies; every GPU verifies the same number of tokens per step, so the step stays in a CUDA graph.
DESIGN.md has the design, the measurements behind each choice, and the time per step by kernel.
volta (--bin volta) runs the engine without the server:
volta bench --model ... --sessions 8 --steps 32 --text tokens.i32 |
throughput with --sessions per GPU; --text feeds real tokens (i32 LE) |
volta eval --model ... --eval <dir> |
the Lucebox eval; --chunk k feeds k tokens per sequence per step |
volta spec --model ... --text tokens.i32 --k 5 --adaptive |
greedy decoding with the draft (VOLTA_SPARK=<checkpoint dir>); --total N for N sessions in all |
volta check, volta experts |
logits and expert kernels against the CPU reference |
| Variable | Effect |
|---|---|
VOLTA_EXPERT_GB |
expert bytes per GPU when packing from the official checkpoint (default 25; the published pack is 27.5) |
VOLTA_GRAPHS |
most step shapes kept as CUDA graphs (the server's default is 4) |
VOLTA_TRACE_STEP |
wraps that step in a profiler range, for nsys --capture-range=cudaProfilerApi |
VOLTA_PROF |
prints where each step's time went |
VOLTA_OVERLAP=0, VOLTA_ATTN=tile5, VOLTA_SPARK_CB2=0 |
one stream, CUDA-core attention, cb3 draft experts |
VOLTA_KV=bf16 |
store the compressed KV cache in bf16 instead of FP4 (3.5x larger) |
- A 256K-context run with 8 head-parallel sessions and the most drafts per step.
- Volta Scout's expert kernel (vlut2: gate/up 31 -> 19 us at one token in the lab, same bits) with the shared-expert
slice moved beside the router (
VOLTA_SHARED_EARLY=1), so the expert pass has the GPU to itself. - The routing chain folded into the router's last block (built in the lab, not yet measured).
- The MoE exchange and the down projection, the remaining per-layer costs above their floors (DESIGN.md).
- A router process in front of the server, so sampling settings change without a restart.
| Path | |
|---|---|
engine/src/volta.rs, engine/src/volta/spark.rs |
the engine: packing, placement, steps, attention, MoE, collectives, CUDA graphs; the draft |
engine/src/kernels/volta.cuh |
the sm_70 kernels |
engine/src/server/volta.rs, engine/src/bin/volta_server.rs |
the batching scheduler and the server |
engine/src/server/telemetry.rs, engine/src/server/dashboard.html |
the admin dashboard's numbers and page |
core/, the rest of engine/ |
model loader, CPU reference, the HTTP server and web UI, from dsx |
plans/ |
the published pack's tier plan and placement, and the routing counts that choose tiers |
tests/ |
standalone kernel tests against reference kernels: expert formats (vmoe_test, vlut_test), dense and expert launch shapes (dense_test, split_test), attention (attn_test), indexer (indexer_test) |
bench/ |
hardware probes and the existing-engine benchmarks |
DeepSeek for the model (MIT) · dsx for the model
loader, CPU reference, server and web UI · Lucebox
for the eval set · the web UI is llama.cpp's, by Georgi Gerganov and the ggml
authors (MIT), vendored in core/webui with its license.
MIT license.