DeepSeek-V4.1-Flash on A100 (sm80)

This repository contains a custom inference runtime for DeepSeek-V4.1-Flash on NVIDIA A100 GPUs. It keeps FP8 dense weights and packed FP4 MoE experts on GPU, uses Triton/CUDA kernels with BF16 tensor cores, and keeps the large Engram tables in host RAM.

Updated 2026-09-16. The numbers below are from the live five-GPU server logs in results/prefix-audit-20260916/.

• 40 backbone layers and 384 experts are distributed across GPUs 2,0,1,3,4 with EP shards 77,77,77,77,76.
• 1,000,000-Token Prefill Demonstrated: Full 1M context cold prefill is empirically verified and benchmarked on 4× A100 80GB (cuda:0,1,2,3) using 4-stage Chunked Pipeline Parallelism and head-accumulated BMM indexer with zero OOM errors.
• CED (Causal Encoder-Decoder) / Decoder SWA Bounded Replay: Intermediate prefill chunks run exclusively through layers 0..20 (Encoder side), generating all global compressed KV and candidate index representations. Decoder layers 21..39 only process the final 128-token SWA window, saving ~47% of prefill compute on long contexts.
• In-GPU Slot Cache & Prefix Replay: Consecutive requests sharing a common prefix (e.g. Claude Code tool use / agentic coding) detect Longest Common Prefix (LCP) in GPU VRAM across active slots (copy_seq), forwarding only suffix tokens (e.g. 500 suffix tokens in 2s vs re-prefilling 128K tokens in 15 minutes). The web dashboard clearly distinguishes Cold Prefill from Prefix Replay with live hit rate and instant ETA.
• Compacted Canonical Snapshot Storage: Host-RAM snapshots deduplicate multi-device GPU mirrors into a single canonical copy per owner, bound SWA windows to 128 rows, and slice Engram history to logical prefix length, slashing host RAM footprint by ~75% and accelerating restore via fast P2P GPU broadcast.
• Jev Mode Implemented: Parallel non-autoregressive structured output engine with hierarchical persistent GPU prefix caching (JevPrefixTree). Replaces serial autoregressive JSON decode with parallel candidate log-probability scoring, achieving up to 2,854× speedup on 30-field extraction (24 ms) and 100% schema consistency with zero output decode tokens.
• Prefix snapshots, multi-anchor reuse, and continuation block replay are implemented.
• DSpark/MTP draft and verification are implemented and verified with --mtp 5 --mtp-device 4.
• For prompts over DSV41_MTP_LONG_PROMPT_LIMIT, the server can unload DSpark and fall back to ordinary decode. This reduces MTP memory pressure but does not remove the full-context cache limit.

DeepSeek-V4.1 supports flexible deployment profiles tailored to specific production workloads—ranging from ultra-low latency, multi-agent interactive coding to 1,000,000-token full-repository analysis.

Best for: Interactive developer workflows (e.g. Claude Code, Cursor, Copilot) where low time-to-first-token (TTFT), fast decode speed, and parallel tool-calling are critical.

By restricting maximum context length to a pragmatic 64K tokens (ample for >99% of development sessions), each A100 GPU frees over 10 GiB of VRAM. This freed memory is dedicated to expanding sequence concurrency to 4 parallel decode slots (max_seqs 5 = 1 prefill scratchpad + 4 decode streams).

#!/usr/bin/env bash
# run_speed_agent.sh: High-throughput 4-slot concurrent decode
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

# Restrict context length to 64K to conserve KV memory
export DSV41_CACHE_INIT_TOKENS=32768
export DSV41_EP_PREALLOC_TOKENS=32768
export DSV41_EXACT_CACHE_GROW=1

# Concurrency: 5 sequence slots (slot 0: prefill scratchpad + slots 1..4: decode)
export DSV41_MAX_SEQS=5

# Acceleration & Caching
export DSV41_EP_COMPACT_XQ=1        # Compressed P2P activation transfers
export DSV41_GPU_SLOT_CACHE=1       # In-GPU slot LCP reuse (0 ms prefill for multi-turn loops)
export DSV41_CED=1                  # Causal Encoder-Decoder (skip layers 21..39 on intermediate chunks)
export DSV41_LOOP_DETECT=1          # Degenerate repetition loop detection & auto-stop

python -m dsv41.serve \
  --ckpt /mnt/ssd/models/DeepSeek-V4.1-Flash-Abliterated \
  --devices 2,3,0,1 \
  --ep --ep-shards 92,95,99,98 \
  --max-seq-len 65536 \
  --max-seqs 5 \
  --host 0.0.0.0 --port 8000 \
  --mtp 0

• Performance Impact: Allows 4 concurrent agent subtasks or parallel tool requests to decode simultaneously without queuing delay, reaching 80–120 tok/s aggregate throughput.

Best for: Multi-agent swarms, parallel code generation benchmarks, high-density serving, and automated test runners.

Restricting maximum context length to 128K tokens (131,072) reduces KV cache memory consumption to just ~1.56 GiB per slot (compared to ~12.5 GiB/slot for 1M context). This dramatic memory reduction allows sequence concurrency to be expanded to 8 parallel decode slots (max_seqs 9 = 1 prefill scratchpad + 8 concurrent decode streams) on 4× A100 80GB GPUs while maintaining 7–11 GiB of VRAM headroom per GPU.

#!/usr/bin/env bash
# run_128k_8slots.sh: High-concurrency 8-stream decode with 128K context
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

# Context sizing (128K horizon)
export DSV41_CACHE_INIT_TOKENS=32768
export DSV41_EP_PREALLOC_TOKENS=32768
export DSV41_EXACT_CACHE_GROW=1

# Concurrency: 8 parallel decode streams (max_seqs 9: slot 0 scratchpad + slots 1..8 decode)
export DSV41_MAX_SEQS=9

# Communication & Cache Acceleration
export DSV41_EP_COMPACT_XQ=1
export DSV41_GPU_SLOT_CACHE=1
export DSV41_PREFIX_DEDUP_MIRRORS=1
export DSV41_CED=1

python -m dsv41.serve \
  --ckpt /mnt/ssd/models/DeepSeek-V4.1-Flash-Abliterated \
  --devices 2,3,0,1 \
  --ep --ep-shards 92,95,99,98 \
  --max-seq-len 131072 \
  --max-seqs 9 \
  --host 0.0.0.0 --port 8000 \
  --mtp 0

• Performance Impact: Serves up to 8 parallel agent subtasks or code generation streams concurrently, scaling aggregate throughput to 110–120 tok/s (119.2 tok/s on 8-stream code generation, 119.9 tok/s peak dashboard decode).

Best for: Ingesting entire code repositories (hundreds of thousands of lines), analyzing whole libraries, or processing massive document corpuses up to 1,000,000 tokens.

At 1M context, standard Transformer implementations exhaust VRAM during KV cache expansion or intermediate activation generation. This profile clamps sequence slots to max_seqs 2 (1 prefill scratchpad + 1 decode slot), minimizing the batch dimension of global compressed KV tables. Combined with CED (Causal Encoder-Decoder), exact cache growth, and host-RAM mirror deduplication, it completes 1M token prefill with 100% stability at 98% GPU memory utilization.

#!/usr/bin/env bash
# run_1m_robust.sh: 1,000,000-token robust milestone configuration
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

# Long-context chunk pipeline configuration
export DSV41_HC_PREFILL_CHUNK=2048
export DSV41_MOE_PREFILL_CHUNK=2048
export DSV41_ENGRAM_PREFILL_CHUNK=512
export DSV41_SPARSE_ATTN_CHUNK=128

# Causal Encoder-Decoder (CED): Binds decoder layers 21..39 to final 128 tokens (cuts compute by ~47%)
export DSV41_CED=1
export DSV41_CED_TAIL_WINDOW=128

# Memory safety guards
export DSV41_EXACT_CACHE_GROW=1     # Strict row allocation prevents runaway VRAM reservations
export DSV41_PREFIX_DEDUP_MIRRORS=1 # Deduplicates multi-GPU mirrors in host RAM (saves 75% host RAM)
export DSV41_GPU_SLOT_CACHE=1       # Multi-turn prefix reuse

# Bound sequence slots to 2 (slot 0: prefill scratchpad + slot 1: decode)
export DSV41_MAX_SEQS=2

python -m dsv41.serve \
  --ckpt /mnt/ssd/models/DeepSeek-V4.1-Flash-Abliterated \
  --devices 2,3,0,1 \
  --ep --ep-shards 92,95,99,98 \
  --max-seq-len 1048576 \
  --max-seqs 2 \
  --host 0.0.0.0 --port 8000 \
  --mtp 0

• Performance Impact: Evaluates 488 pipelined micro-chunks across 4× A100 GPUs (1,000,000 tokens) in ~95 minutes without crashing or encountering out-of-memory errors.

Best for: General software development, multi-turn Claude Code workflows, and 2-stream concurrent serving.

Provides a balanced operating point with 2 concurrent decode slots (max_seqs 3), full 1M context potential, in-GPU slot cache acceleration, dedicated B=1 CUDA graph speed, and 7–10 GiB of free VRAM headroom per GPU.

# Launch with bundled production script (max-seqs=3, 2 decode slots, 1M context ready)
./run_server_batched.sh

• Performance Impact: Sustains single-stream generation at 51.1 tok/s, scales to ~75–80 tok/s aggregate under 2 concurrent requests, and preserves ample memory safety margins.

Best for: Highly interactive single-stream terminal chats and agentic thought generation where minimizing user-perceived token latency is paramount.

In a 4-GPU configuration (--devices 2,3,0,1), DeepSeek's 3-layer Multi-Token Prediction head (DSpark) lives on the last GPU (cuda:1). Because DSpark and the output head require ~15.5 GiB of VRAM, hosting the normal 96+ experts on cuda:1 would cause an out-of-memory error.

The Solution: Rebalance expert parallelism shards to --ep-shards 100,100,100,84:

• cuda:2, cuda:3, and cuda:0 each host 100 experts (+4 experts over even split, +5.1 GiB).
• cuda:1 hosts only 84 experts (12 fewer experts, freeing ~15.4 GiB of VRAM).
• DSpark's 3 blocks, draft KV ring, and LM head fit comfortably within the freed memory.

# Launch 4-GPU MTP speculative decoding (85-93 tok/s single-stream)
./run_mtp_4gpu.sh

• Performance Impact: Proposes 5 draft tokens per step (--mtp 5). With an average acceptance rate of 1.79 drafts/step, generation advances by 2.79 tokens per verified step, accelerating single-stream throughput from 63.3 tok/s to 84.7–93.0 tok/s (~1.35×–1.45× speedup).

curl http://127.0.0.1:8000/health
curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"deepseek-v4.1-flash","messages":[{"role":"user","content":"What is the height of Tokyo Tower?"}],"max_tokens":256,"temperature":0.6,"stream":true}'

# Jev Mode: Parallel Non-Autoregressive Structured Output (Zero decode tokens)
curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "deepseek-v4.1-flash",
    "prompt": "The customer says the service is too expensive and they are considering cancelling unless somebody contacts them today.",
    "jev": true,
    "schema": {
      "sentiment": ["positive", "neutral", "negative"],
      "churn_risk": ["low", "medium", "high"],
      "urgency": ["low", "medium", "high"],
      "needs_human": [true, false]
    }
  }'

The server provides /health, /v1/models, /v1/chat/completions, /v1/completions, and /dashboard (plus /api/metrics). When "jev": true is passed with "schema": {...}, the engine runs in Jev Mode, extracting all fields concurrently via candidate log-probability scoring over the hierarchical persistent prefix cache. If "stream": true is set, Jev mode emits keep-alive heartbeat SSE comments (: keep-alive\n\n) during prefill to prevent client gateway timeouts, followed by OpenAI-compatible SSE chunks and data: [DONE].

Navigate to http://127.0.0.1:8000/dashboard in any web browser for live server telemetry:

• Concurrent Context Streams & Typewriter Output: Real-time generation streaming across parallel context slots with live token rate and typewriter animation.
• Live Prefill & Chunk Monitor: Real-time tracking of chunk progress, token processed count, compute rate, and completion ETA during large-context prefill.
• Throughput Benchmarks: Rolling 10-minute Peak (Max) and Min decode speeds alongside 24h rolling average time-series.
• Dynamic KV Cache & Capacity: Live allocation breakdown of compressed KV rows and index keys across GPU devices.
• GPU VRAM & Utilization (GPUs 0–3): Live per-device VRAM allocation (GiB / 80 GiB) and core utilization (%).
• Host RAM Prefix Cache: Track persistent prefix cache entries and host RAM/tmpfs memory footprint.
• 100% Self-Contained: Offline HTML5 Canvas rendering with zero external scripts or CDN dependencies.

The runtime restores mutable window KV, compressed KV, index keys, compressor state, Engram history, and DSpark state from a prefix snapshot. Block replay uses the model continuation path; ordinary decode continues to use the EPRuntime CUDA-graph path. Multi-anchor replay never skips a pending anchor.

The current snapshot epoch is fullstate-v5-index-engram-dspark. Snapshots from older epochs are intentionally skipped because they lack required persistent state.

Useful log lines:

[prefix-cache] LCP-HIT old=16,264 new=16,398 lcp=16,264 base=16,264 replay=134
[prefix-snapshot] RESTORE slots=98
[prefix-replay-block] ... block=512 ... tok_s=169.4
[prefix-replay-mtp-tail] main_hidden=OK shape=(1, 1, 15360)
[prefill-bench] mode=LCP-HIT new=134 reused=16,264 total=16,398 ...

These measurements used five-GPU EP, block size 512, MTP=5, and multi-anchor snapshots. Replay/prefill throughput is distinct from generated decode throughput.

Full-request examples:

[prefill-bench] mode=LCP-HIT new=8,320 reused=8,072 total=16,392 time=24.805s new_tok_s=335.4 effective_tok_s=660.8
[prefill-bench] mode=LCP-HIT new=134 reused=16,264 total=16,398 time=0.999s new_tok_s=134.1 effective_tok_s=16,410.5

At 117,896 tokens, MTP fallback and DSpark unloading were triggered, but the compressed/index cache eventually exhausted GPU memory and the API returned HTTP 507. This is a persistent full-context capacity limit, separate from short prefix-HIT replay performance.

[mtp] enabled drafts=5 verify_rows=6
[generate] STOP=eos produced=2 mtp_accept=2/1
[prefix-replay-mtp-tail] main_hidden=OK shape=(1, 1, 15360) dtype=torch.bfloat16 device=cuda:4

Full 1M-token (1,048,576 max logical) cold prefill has been successfully demonstrated and benchmarked on 4× NVIDIA A100 80GB PCIe GPUs (cuda:0, 1, 2, 3) using 4-stage Chunked Pipeline Parallelism ($C=2048$) and head-accumulated BMM indexer scoring.

• Total Tokens Prefilled: 1,000,000 tokens (488 micro-chunks of 2,048 tokens + final 496 tokens)
• Total Elapsed Time: 5,729.20 s (95.49 min / ~1.59 hours)
• Cumulative Average Throughput: 174.54 tok/s
• Peak VRAM Footprint: [79.11, 76.00, 78.86, 77.38] GiB (over 97% capacity on 80 GiB A100s)
• Stability: Zero CUDA out-of-memory (OOM) errors, zero kernel crashes, 100% completed.

The following measurements evaluate cold prefill performance across varying context lengths on 4× A100 80GB GPUs:

A critical observation from the benchmarks is why single-request short-context prefill reaches ~600 tok/s, while ultra-long context prefill operates at ~396 tok/s initially and scales down to ~175 tok/s cumulative average at 1,000,000 tokens:

1. Dynamic Sparse Attention (DSA) Indexer $T$-Scaling ($O(T)$ Key Retrieval):

   • In DeepSeek-V4.1, attention compute is sparse: each query token only attends to Top-$K$ key tokens ($O(K)$ attention computation per query, which is strictly $O(1)$ relative to sequence length $T$).
   • However, to determine which $K$ keys to attend to, the DSA Indexer module must compute similarity scores between query index vectors and all historical compressed key tokens: $$\text{Scores}{\text{index}} = Q{\text{idx}} K_{\text{idx}}^T \quad (Q_{\text{idx}} \in \mathbb{R}^{S \times D}, K_{\text{idx}} \in \mathbb{R}^{T \times D})$$
   • When context is short ($T \le 8,192$), $K_{\text{idx}}$ is small (< 2 MB), and the index scoring matrix multiplication completes in microseconds with negligible FLOPs.
   • When context reaches 1,000,000 tokens ($T = 10^6$), each incoming micro-chunk ($S=2,048$) must score against the full historical key set of up to 1M tokens across 32 index heads ($D=128$). Even with GPU-accelerated BMM, the memory bandwidth required to scan 1M keys and the dot-product FLOPs scale linearly with $T$ ($O(S \cdot T)$).
   • Consequently, chunk latency smoothly scales from 5.17 s per chunk (396 tok/s) at $T=32\text{K}$ to 18.2 s per chunk (112.5 tok/s) at $T=983\text{K}$.

2. Monolithic GEMM Saturation vs. Chunked Pipelined Micro-Batches:

   • In short prompts ($T \le 8,192$), the full sequence is fed into monolithic GEMM kernels ($M=2048, 4096, 8192$). Large matrix dimensions maximize Tensor Core arithmetic intensity and saturate all 108 Streaming Multiprocessors (SMs) on the A100, reaching peak theoretical efficiency (~600 tok/s).
   • At 1M tokens, a monolithic forward pass is physically impossible: activation memory alone would exceed 100 GiB, immediately causing an out-of-memory crash.
   • Prefill must therefore be sliced into 2,048-token micro-chunks across a 4-stage pipeline. Slicing prevents activation blowup, but micro-chunk execution incurs pipeline warm-up/drain bubble overhead and operates at slightly lower Tensor Core occupancy than an 8,192 monolithic matrix.

3. 12.5 GiB Static Cache & HBM2 Saturation (>97% VRAM):

   • Supporting 1M tokens requires pre-allocating the full compressed KV cache, window KV cache, and index keys, occupying 12.5 GiB of VRAM per GPU. Total allocated GPU memory reaches 76.0 ~ 79.1 GiB out of 80 GiB (>97% capacity).
   • Operating near physical memory capacity eliminates L2 cache residency for key tables and places continuous demand on the HBM2 memory controller bus, moderating throughput compared to small cache allocations.

1. Head-Accumulated BMM Indexer (dsv41/model.py): Standard indexer implementations perform full-tensor contraction (torch.einsum("bshd,btd->bsht", ...)), which materializes an intermediate tensor of shape (1, S, 32, T). At $T=200,000$, this single tensor required 3.31 GiB of transient activation VRAM, causing an OOM crash. At $T=1,000,000$, it would have required > 16.5 GiB. We re-engineered the indexer to iterate over index heads sequentially using torch.bmm, reducing peak intermediate memory by 32× to a constant 256 MB, completely eliminating indexer OOMs.

2. Zero-Host-Sync GPU Vector MoE Dispatch (dsv41/moe_kernels.py): Eliminated host-side CPU sorting (argsort) in MoE dispatch. Token-to-expert mapping is now performed via pure CUDA tensor operations (GroupedPairs), dropping cudaStreamSynchronize waiting time from 5.5 s to 0.004 s per chunk (>1,000× scheduling speedup).

3. 4-Stage Chunked Pipeline Parallelism (forward_pipelined): Distributed 40 transformer layers across 4 GPUs (10 layers per GPU: Stage 0 = layers 0–9 on cuda:0, Stage 1 = layers 10–19 on cuda:1, Stage 2 = layers 20–29 on cuda:2, Stage 3 = layers 30–39 on cuda:3). Dedicated inter-device P2P CUDA streams and pre-allocated CUDA events overlap activation transfers with stage computations, keeping all 4 GPUs actively computing without CPU blocking.

4. Causal Encoder-Decoder (CED) / Decoder SWA Bounded Replay (dsv41/model.py, dsv41/engine.py): DeepSeek-V4.1's architectural asymmetry partitions the 40 layers into an Encoder section (layers 0–20) and a Decoder section (layers 21–39). All global compressed KV tables (kv_source_layer_ids: [2, 8, 14, 20]) and candidate representations (candidate_source_layer_id: 20) originate exclusively from the Encoder side. Decoder layers 21–39 feature only 128-token Sliding Window Attention (SWA) and do not produce any global KV state. By processing intermediate prefill chunks only through layers 0–20 and bounding layers 21–39 to the final 128 tokens, prefill layer-tokens drop from $40 \times S$ to $21 \times S + 19 \times 128$, cutting compute by ~47%.

Standard Transformer architectures execute every prompt token through all layers:

$$\text{Tokens } [0 \dots S] \longrightarrow \text{Layer } 0 \longrightarrow \text{Layer } 1 \longrightarrow \dots \longrightarrow \text{Layer } 39$$

In DeepSeek-V4.1-Flash, the layer topology is partitioned as follows:

• Encoder Side (Layers 0..20):
  • Contains all KV source layers ([2, 8, 14, 20]) that populate compress_kv and index_k.
  • Layer 20 generates candidates for dynamic sparse attention across the entire prompt.
• Decoder Side (Layers 21..39):
  • ratio = 1 (no downsampling/compression).
  • Attention is strictly local 128-token Sliding Window Attention (SWA).
  • Generates zero global KV state; tokens older than 128 positions are permanently evicted from the SWA buffer during decode.

Prompt Tokens (e.g. 1M or 128K)
      │
      ▼
┌────────────────────────────────────────────────────────┐
│  Stage 1: Causal Encoder (Layers 0 .. 20)              │
│  - Evaluates full prompt (all S tokens)                │
│  - Populates global compress_kv & index_k             │
│  - Produces global topk candidates at Layer 20         │
└────────────────────────────────────────────────────────┘
      │
      │ (Intermediate chunks m < M-1 complete here)
      ▼
┌────────────────────────────────────────────────────────┐
│  Stage 2: Bounded Decoder SWA (Layers 21 .. 39)        │
│  - Tail Window: Slices ONLY final W=128 tokens         │
│  - Intermediate tokens completely skip layers 21..39   │
│  - Populates local SWA window_kv_cache for decode      │
│  - Generates initial next-token logits                 │
└────────────────────────────────────────────────────────┘

For prompt length $S$ (e.g. 128,000 tokens) with window $W = 128$:

• Standard Forward: $40 \times 128,000 = 5,120,000$ layer-tokens
• CED Forward: $21 \times 128,000 + 19 \times 128 = 2,688,000 + 2,432 = 2,690,432$ layer-tokens
• Total Compute Savings: 47.45% FLOPs saved

Controlled by DSV41_CED=1 (enabled by default) and DSV41_CED_TAIL_WINDOW=128.

For multi-turn agentic workflows (e.g. Claude Code tool use), successive requests share an extensive common prefix (often 95%–99.9% identical tokens). Re-prefilling 128K context for every tool call wastes minutes of compute time.

1. In-GPU Slot Reuse (_find_best_gpu_slot):
   • The engine tracks token sequences across active GPU slots (0 through max_seqs - 1).
   • Upon a new request, it computes the Longest Common Prefix (LCP) against all slots.
   • If an existing slot contains a matching prefix, the runtime copies slot state directly in GPU VRAM via copy_seq (< 0.05 ms) and forwards only the suffix delta on slot 0.
2. Cold Prefill vs. Prefix Replay Separation:
   • The Web Dashboard (/dashboard) and /api/metrics dynamically distinguish:
     • PHASE: PREFIX REPLAY [GPU HIT / HOST REPLAY] (green/cyan badge): Displays LCP tokens, hit rate %, suffix tokens, and instant ETA.
     • PHASE: COLD PREFILL (orange badge): Displays full prompt chunk progress.
   • Eliminates misleading "15-minute" ETA estimates when 99.8% of the prompt is already cached.

1. Mirror Deduplication:
   • DeepSeek-V4.1 mirrors shared attention tables (compress_kv and index_k) across all 4 GPUs.
   • Prior snapshot engines dumped all 4 mirrors separately into host RAM, resulting in 122 GiB of bloat across 19 entries.
   • The snapshot manager now records only 1 canonical copy per owner in host RAM and tmpfs, broadcasting GPU-to-GPU via PCIe/NVLink upon restore (75% RAM reduction).
2. Dynamic Cache & History Slicing:
   • NgramHashState.cache is sliced from 1,000,000 tokens down to the logical prefix length.
   • window_kv_cache is bounded to 128 rows.

Traditional structured JSON extraction with LLMs serializes data generation into dozens or hundreds of autoregressive decode steps (e.g. generating {", field names, quotes, colons, commas, and formatting syntax). Each decode step requires an independent forward pass, causing high latency (8 to 68+ seconds for 3 to 30 fields) and vulnerability to formatting errors or premature EOS stops.

Jev Mode (dsv41/jev.py) replaces serial token generation with parallel non-autoregressive candidate scoring over a hierarchical persistent prefix cache (JevPrefixTree). Instead of generating JSON syntax, the runtime treats schema fields as independent queries over the shared prompt representation and scores candidate token log-probabilities directly from next-token logits in parallel.

[Prefix Tree Root]
       │
[Level 1: Jev System Prompt]  <-- Prefilled at startup, permanently retained in GPU VRAM (0 ms prefill)
       │
[Level 2: Schema Prefix]      <-- Cached separately per schema on GPU (Cache-hit latency: 0.08 ms)
       │
[Level 3: Request Input]      <-- User request tokens appended once -> Shared Request KV (~11 ms)
       │
 ┌─────┴───────────────────────┬────────────────────────┐
[Level 4: Field 1 Query]     [Level 4: Field 2 Query]  [Level 4: Field N Query] (Uniform 5-tok queries in parallel batch)
 │                             │                         │
[Level 5: Candidate Logits]   [Level 5: Candidate Logits] [Level 5: Candidate Logits]
 │                             │                         │
 └─────────────────────────────┴─────────────────────────┘
                               │
                       [JSON Assemble] <-- Native Python dict directly constructed (0 decode tokens)

1. Level 1 (Jev System Prompt): Fixed system prompt is prefilled once at server startup and permanently retained in GPU VRAM. It is never prefilled again.
2. Level 2 (Schema Prefix): Schema definitions are compiled into concise query definitions and cached separately on GPU. Subsequent requests with the same schema hit in 0.08 ms with zero prefill overhead.
3. Level 3 (Request Input): Request input tokens are prefilled once onto the active KV cache, forming the Shared Request KV.
4. Level 4 (Field Queries): Field-specific suffixes (\nField {i}:, uniform 5 tokens each) are branched across the batch dimension without duplicating prefix storage in memory (GPU-to-GPU broadcast in < 0.01 ms). All fields execute in a single batched forward pass.
5. Level 5 (Candidate Scoring): Candidate values for categorical/boolean/enum fields are scored directly from output logits via teacher-forced log-probabilities ($S = \log P(\text{cand} \mid \text{prefix})$). The candidate with the highest log probability is chosen. Zero autoregressive decode tokens are generated.
6. Direct Assembly: Assembled directly into a typed Python dictionary / JSON object. Syntax errors and formatting failures are mathematically eliminated.

• Input Prompt: "The customer says the service is too expensive and they are considering cancelling unless somebody contacts them today."
• Schema:
  • sentiment: ["positive", "neutral", "negative"]
  • churn_risk: ["low", "medium", "high"]
  • urgency: ["low", "medium", "high"]
  • needs_human: [true, false]
• Jev Mode Output:

  {
    "sentiment": "negative",
    "churn_risk": "high",
    "urgency": "high",
    "needs_human": true
  }

• Accuracy: 100% match with ground truth model classification.
• Total Latency: 23.98 ms
  • Hierarchical Cache-Hit Latency: 0.080 ms
  • Request Input Prefill: 11.40 ms
  • Candidate Scoring (all 4 fields in parallel): 12.50 ms
• Output Tokens Generated: 30 tokens (extracted and assembled in parallel without autoregressive decode)
• Effective Throughput: 1,251 tokens/second

Measured using python3 -m dsv41.bench_jev on 4× NVIDIA A100 80GB PCIe GPUs:

1. O(1) Latency Scaling & Massive Throughput: While normal autoregressive generation scales linearly with field count (2.0 s → 4.2 s → 9.8 s → 30.7 s), Jev Mode execution latency remains flat at 24.0 ms from 3 up to 30 fields, scaling effective structured output throughput from 917 tok/s up to over 12,980 tok/s.
2. 408× Speedup: On 30 fields, Jev Mode reduces latency from ~9.8 seconds down to 24 milliseconds.
3. 500× Speedup on 100 Fields: Normal generation takes over 30 seconds serializing 898 tokens, whereas Jev Mode extracts and tokenizes all 100 typed fields in 61.5 ms (~13,000 tok/s effective throughput).

See examples/jev_mode/ and examples/benchmarks/bench_jev_structured.py for runnable code and live demos.

Measured using examples/benchmarks/bench_decode_throughput.py on 4× NVIDIA A100 80GB PCIe GPUs:

Measured on 4× NVIDIA A100 80GB GPUs (--devices 2,3,0,1 --ep-shards 100,100,100,84):

Measured using examples/benchmarks/bench_code_generation.py on 4× NVIDIA A100 80GB PCIe GPUs across algorithmic, data structure, and system utility tasks:

• Average Single-Stream Coding Speed: 52.70 tok/s (consistent ~50–53 tok/s across all tasks)
• Python AST Syntax Pass Rate: 80.0% (100% on completed code blocks)
• Functional Unit Test Pass Rate: 80.0% (inline test suites executed and passed in sandbox)

Measured using examples/benchmarks/bench_prefix_cache_hit.py simulating a multi-turn developer session with a shared 1,367-token context (codebase architecture, tool definitions, rules):

DeepSeek-V4.1 can be used as a drop-in local inference backend for Anthropic's official Claude Code CLI (claude), providing an entirely private, self-hosted coding assistant with multi-turn prefix cache acceleration.

flowchart LR
    subgraph Client ["Developer Workstation"]
        CC["Claude Code CLI<br/>(ANTHROPIC_BASE_URL)"]
    end

    subgraph Proxy ["LiteLLM Translation Layer"]
        LL["LiteLLM Proxy<br/>(:8101)<br/>Anthropic Messages ↔ OpenAI Chat"]
    end

    subgraph Backend ["Local GPU Cluster"]
        DS["DeepSeek-V4.1 Engine<br/>(:8000)<br/>EP + Dynamic Compressed KV + LCP Cache"]
    end

    CC -->|Anthropic API /v1/messages| LL
    LL -->|OpenAI API /v1/chat/completions| DS

1. Start LiteLLM Proxy: Use the tested configuration in examples/claude_code/dsv41-litellm.yaml:

   litellm --config examples/claude_code/dsv41-litellm.yaml --host 127.0.0.1 --port 8101

2. Verify Connectivity & Tool Calling: Run the verification suite in examples/claude_code/verify_litellm.py:

   python3 examples/claude_code/verify_litellm.py

   (Verifies health, model translation, LCP prefix cache hit, and Anthropic tool use format).

3. Launch Claude Code: Run the pre-configured production wrapper in examples/claude_code/claude-dsv41.sh:

   bash examples/claude_code/claude-dsv41.sh

Claude Code sessions send growing conversation histories that include system instructions, codebase rules, directory trees, and previous tool outputs. DeepSeek-V4.1's Longest Common Prefix (LCP) cache matches these shared prefixes in GPU VRAM across consecutive turns:

• Turn 1: Cold prefill of project context (~2,000–8,000 tokens).
• Subsequent Turns: Reuses up to 99.5% of cached tokens directly on GPU; the server evaluates only the new tool output or user prompt in < 1 second.

See examples/claude_code/README.md for detailed configuration, model aliasing, and troubleshooting.

Complete guides, runnable scripts, and interactive tools are located in the examples/ directory:

These older measurements use different GPU counts, cache lengths, and batch sizes and must not be compared directly with the long-context table above.

Known limitations are no continuous batching, no vision input, and delayed SSE output. Full 1M context prefill is supported via 4-GPU chunked pipeline parallelism.