Shrink the cache, keep the brains.
R-KV discards repetitive tokens on-the-fly, delivering full-accuracy reasoning with only a fraction of the memory.

｜🌏 Page | 📖 Paper｜

• Apply R-KV in GPT-OSS
• Integrate R-KV with VeRL
• Harden R-KV benchmark coverage in vLLM, Nano-vLLM and SGLang (GPU smokes in tests/smoke/, throughput bench + raw artifacts in results/, CPU tests in CI)
• Performance-harden the production serving ports (vLLM + SGLang) (in-graph decode, async scheduling, batched cross-layer scoring, bounded-footprint eviction, TP + DP; memory-bound throughput win — see vLLM/benchmark/ and SGLang/benchmark/)
• Extend dataset to GPQA, liveCodeBench
• Apply R-KV in QwQ
• Expand Qwen-3 evaluation coverage

• 🚀 [26/07/17] vLLM R-KV port — major performance optimization, now on main. The vLLM (v0.25.1) port is performance-hardened end-to-end: in-graph PIECEWISE cudagraph decode (no --enforce-eager), async scheduling, batched cross-layer redundancy scoring, and bounded-footprint block freeing (FREE_BLOCKS, returns evicted KV to the allocator). This turns the concurrency benefit into a real memory-bound throughput win — R-KV holds far more requests in flight than Full-KV as the KV pool shrinks (+17 % to +32 % tok/s at gpu_mem 0.40–0.25) while staying lossless at budget=512 on GSM8K — plus tensor + data parallelism (DP scales 7.6× on 8× H100; every TP rank evicts the identical set). Shipped as a clean patch-not-fork layout (vLLM/scripts/apply_rkv.sh). Resolves the long-standing community request in #3. See vLLM/benchmark/ (RESULTS.md, RESULTS_stress.md, RESULTS_tp.md, RESULTS_dp.md) and vLLM/docs/OPTIMIZATIONS.md.
• 🚀 [26/07/13] SGLang R-KV port — faster, still lossless. The decode-time port is now performance-hardened: a fused Triton redundancy kernel, in-graph CUDA-graph decode, batched cross-layer scoring, two-phase compaction, and compression-aware admission — plus tensor + data parallelism (DP scales 7.8×, TP validated on 8× H100 via a cross-rank eviction-score all-reduce). Accuracy stays lossless at budget=512 on GSM8K (on par with Full-KV) while throughput sits within a few percent of the fair Full-KV baseline. See SGLang/benchmark/ (RESULTS.md, RESULTS_dp.md, RESULTS_tp.md) and SGLang/docs/OPTIMIZATIONS.md.
• 🚀 [26/07/02] The SGLang (v0.5.14) R-KV port is complete. Decoding-time KV-cache compression with true physical eviction on the FlashInfer decode path, validated on Qwen2.5-Math-7B (H100): accuracy on par with the baseline (95% GSM8K), server-side batching (eval.py --concurrency N), and plain data parallelism (DP=N ./launch_server.sh rkv 256, throughput scales up to 7.8× on 8× H100). Code in SGLang/rkv/ (algo.py, integration.py) plus the wiring patch SGLang/patch/rkv-sglang-0.5.14.patch; design/implementation notes in SGLang/docs/DESIGN.md and SGLang/docs/IMPLEMENTATION.md; benchmarks + reproduce steps in SGLang/benchmark/ (RESULTS_math7b.md, RESULTS_dp.md, RESULTS_a100_n100.md).
• ⚠️ [26/07/02] If you computed GSM8K numbers with this repo before commit e9f54c45, please rerun them. The shipped data/gsm8k.jsonl carried a stale generation field from an old experiment and eval_math.py preferred it over the fresh run's output, freezing GSM8K scores at ~40% regardless of method or budget (issues #26/#23). MATH and AIME24 were unaffected. All four serving integrations (vLLM, SGLang, Nano-vLLM, Mini-SGLang) plus the HuggingFace path are now GPU-validated on A100 with R-KV on/off — see tests/smoke/ and results/validation-2026-07-02-a100/.
• 🚀 [25/05/29] We are pleased to announce the release of R-KV, a highly efficient decoding time KV Cache compression method to serve reasoning Models.

Use the following command to install the minimal required dependencies for the HuggingFace implementation:

pip install -r requirements.txt

For the SGLang port (v0.5.14), apply R-KV to a pinned SGLang checkout, then install the verified stack:

cd SGLang
bash scripts/apply_rkv.sh                  # clone pinned SGLang + drop in rkv/ + apply the wiring patch
pip install -e "sglang-src/python" --extra-index-url https://docs.sglang.ai/whl/cu129/
pip install -r requirements-rkv.txt        --extra-index-url https://docs.sglang.ai/whl/cu129/

See SGLang/README.md for the pinned environment and the step-by-step reproduce.

For lightweight reference ports, see Nano-vLLM/ and Mini-SGLang/. Nano-vLLM/ is a ~1.2k-line paged-KV inference engine with R-KV plugged into the attention path (enforce_eager=True required); Mini-SGLang/ carries the R-KV algorithm and a paged-cache integration helper under python/minisgl/compress/, with the attention-layer wiring documented in Mini-SGLang/docs/RKV.md.

For the vLLM port (v0.25.1), R-KV ships as a small wiring patch over a pinned upstream checkout — the same patch-not-fork layout as SGLang — not a vendored tree:

cd vLLM
scripts/apply_rkv.sh            # clone pinned vLLM v0.25.1 + drop in rkv/ + apply the wiring patch
pip install -e vllm-src         # source build (needs CUDA + a GPU)

Compression activates automatically once budget and buffer are both > 0 (env vars VLLM_V1_R_KV_BUDGET / VLLM_V1_R_KV_BUFFER); do not pass --enforce-eager — R-KV auto-selects PIECEWISE cudagraph. See vLLM/README.md and vLLM/docs/REPRODUCE.md.

If you're using Hugging Face, we default to flash attention to speed up attention computation:

model = AutoModelForCausalLM.from_pretrained(
    "model_name_or_path",
    attn_implementation="flash_attention_2",
)

You need to build the dependencies of the evaluation toolkit separately:

cd HuggingFace
cd evaluation/latex2sympy
pip install -e .
cd ..
pip install -r requirements.txt

Before running the scripts, you need to build the rkv package:

cd HuggingFace
pip install -r requirements.txt
pip install -e .

Use the following command to run R1-like models with R-KV on math benchmarks:

bash scripts/run.sh

Or you could use the code scripts:

export CUDA_VISIBLE_DEVICES=0
cd HuggingFace

python3 ./run_math.py \
--dataset_path ./data/aime24.jsonl \
--save_path ./outputs/output.jsonl \
--model_path deepseek-ai/DeepSeek-R1-Distill-Llama-8B \
--max_length 32768 \
--eval_batch_size 1 \
--method rkv \
--kv_budget 1024 \
--window_size 8 \
--mix_lambda 0.1 \
--divide_method step_length \
--divide_length 128 \
--do_sample \
--temperature 0.6 \
--top_p 0.95

For paper-style candidate generation, additionally set --num_return_sequences 64. Add --use_chat_template when evaluating DeepSeek-R1-distill (or other chat) models: they only enter their trained <think> reasoning format inside the chat template.

To evaluate benchmark results, simply run:

bash examples/eval.sh

The results will be saved in the outputs directory.

The vLLM/ and SGLang/ directories are decode-time drop-ins with one-command launchers and matching eval harnesses; full reproduce steps live in each port's README.md and benchmark/.

vLLM (v0.25.1) — after scripts/apply_rkv.sh + pip install -e vllm-src, from vLLM/benchmark/:

./prepare_data.sh
# offline GSM8K: accuracy + throughput + physical compaction count
VLLM_V1_R_KV_BUDGET=256 VLLM_V1_R_KV_BUFFER=128 python eval.py --n 200 --label rkv_b256
# ...or serve an OpenAI-compatible endpoint with R-KV on:
./launch_server.sh rkv 256

SGLang (v0.5.14) — after scripts/apply_rkv.sh, from SGLang/:

bash benchmark/prepare_data.sh
MODEL=/path/to/Qwen2.5-Math-7B-Instruct bash benchmark/launch_server.sh rkv 512   # serve with R-KV
python3 benchmark/eval.py --n 20 --concurrency 8 --label rkv_b512                 # drive the served eval

Memory-pressure throughput (where R-KV wins): vLLM/benchmark/RESULTS_stress.md and SGLang/benchmark/RESULTS_stress.md.

CPU-only unit tests cover the R-KV algorithm port in the lightweight reference implementations:

# Nano-vLLM port — pre-trigger no-op, post-trigger shape + trailing-window
# preservation, and cross-port equality against Mini-SGLang.
python Nano-vLLM/tests/test_rkv_algorithm.py

# Mini-SGLang port — same algorithm tests plus disabled-compressor /
# drop_request / drain_pending_free_slots integration coverage.
python Mini-SGLang/tests/test_rkv_algorithm.py

# SGLang port — algorithm parity, compaction/lifecycle/batch integration, and
# cross-repo bit-level parity against this repo's rkv/ reference.
python SGLang/tests/test_rkv_algo.py
python SGLang/tests/test_rkv_integration.py
python SGLang/tests/test_cross_repo_parity.py

All run on CPU without a GPU and exercise the exact R1KV class used at runtime. The Nano-vLLM and Mini-SGLang tests also run in CI on every push (.github/workflows/cpu-tests.yml).

GPU smoke tests for all four serving integrations (vLLM, Nano-vLLM, Mini-SGLang, SGLang) plus the HuggingFace monkeypatch live in tests/smoke/ — each generates with R-KV on/off and applies a lexical health check that catches garbled or looping output. Raw outputs, accuracy metrics, and vLLM throughput benchmarks from the validated A100 runs are checked in under results/.

Large language models that rely on chain-of-thought (CoT) or self-reflection can crack tough reasoning tasks—but at the cost of very long outputs that bloat the key–value (KV) cache during inference.
Traditional cache-compression schemes, tuned for long prompts, tumble on these generated traces, keeping only ~60 % of the original accuracy when restricted to 10 % of the cache.

R-KV — Redundancy-aware KV-cache compression for Reasoning models — solves this by ranking tokens on-the-fly for both importance and non-redundancy, retaining only the informative, diverse ones.

• 10 % cache → ≈ 100 % accuracy
• 16 % cache → 105 % accuracy
  (noise reduction even nudges performance above the full cache)
• 90 % memory saved and 6.6 × throughput during long CoT generation
• Consistent wins over all prior baselines on two math-reasoning benchmarks

At 16 % cache, noise removal even boosts accuracy to 105 % of the full baseline.

• Up to 90 % KV-cache memory savings with zero—sometimes negative—accuracy loss
• Plug-and-play: a lightweight wrapper for any autoregressive LLM
• Training-free: drop straight into inference or RL roll-outs—no finetuning required

Chain-of-thought (CoT) and self-reflection unlock impressive reasoning, but they explode the key–value (KV) cache.
A single DeepSeek-R1-Distill-8B run on a tough math problem can:

• Generate 32 000 tokens
• Load 15.5 GB of weights
• Allocate 4.1 GB of KV just to remember its own musings

Existing compression tools focus on long prompts and falter on long generations—often pruning the wrong tokens because redundant self-checks still attend heavily to themselves.

Redundancy-aware KV Cache Compression for R1-Style models (i.e., R-KV) is the first production-level KV Cache Compression Method for serving R1-like models.

Large-language models (LLMs) waste a surprising amount of memory on redundant key/value (KV) tokens during long-form reasoning.
R-KV tackles this by compressing the KV cache on-the-fly while the model is decoding, keeping only tokens that are important and non-redundant.

Our method surpassed baselines by a large margin in challenging math benchmarks, and surprisingly, even outperformed full KV.

We benchmark two distilled DeepSeek-R1 variants:

Sampling: temperature 0.6, top-p 0.95, 64 candidates per problem; we report pass@1.

For the HuggingFace runner, B_buffer = 128 maps to --divide_method step_length --divide_length 128. --kv_budget is the retained KV budget (B_budget). divide_method=newline is a convenience heuristic and was not the fixed-buffer setting used for the main reported results.

The released HuggingFace implementation performs decoding-time compression. It should not be used as a prefill-stage long-prompt compressor or TTFT optimization without adding a separate prefill compression pass.

• FullKV – no compression, upper-bound quality.
• SnapKV – adapted from prompt-time to decode-time by compressing every 128 generated tokens with the same budgets as R-KV.
  Head-layer-budget schedule (e.g. PyramidKV, HeadKV) are orthogonal and omitted here.

Pass@1 vs. KV-cache budget on MATH-500 and AIME-24.
R-KV attains parity with 10–34 % cache and even beats the FullKV baseline at 10 %.

At 16 % (Llama-8B) or 33 % (Qwen-14B) cache, R-KV reaches 105 % of FullKV accuracy—evidence that trimming redundant tokens can improve reasoning quality.

R-KV allocates two fixed-size buffers—one for the retained KV cache and one for freshly generated tokens.
Because these buffers stay the same size regardless of sequence length, memory usage remains constant, unlike FullKV, whose memory grows linearly with the sequence.

R-KV keeps two small, fixed-size buffers:

b = batch · L = #layers · H = #heads · d = head-dim

An optional query cache keeps only the last α query states.

R-KV adds a lightweight scoring pass for importance and redundancy, but the extra FLOPs are quickly offset by having to attend over a much smaller, compressed KV cache.
As sequences get longer, the cost/benefit curve tips even further in R-KV’s favor (details in the same appendix).

For long sequences (B_full ≫ B_budget), the tiny cache more than offsets scoring overhead.

We measured both memory savings and end-to-end throughput (see Table efficiency in the paper):

• Batch = 1: R-KV already edges out FullKV in tokens-per-second, showing that reduced attention cost outweighs the scoring overhead.
• Larger batches: The real win comes from compression—smaller caches let us pack far more sequences into GPU memory, multiplying throughput.

TPS = tokens per second. Throughput scales almost linearly with batch until compute saturation (~128 sequences on an A100).

Full benchmarking scripts and raw logs are provided in below table.

• Constant memory → run much longer sequences without OOM.
• Tiny attention window → lower FLOPs despite a lightweight scoring pass.
• Bigger batches → up to 13 × more sequences in parallel and 9 × higher tokens/s than FullKV.

The figure below shows which tokens are picked by R-KV and the pure-attention baseline SnapKV at the same decoding step.
Grey = not selected  |  Light orange → Dark red = selected tokens (deeper red = chosen by more attention heads)

• Broader Coverage
  With its hybrid score (attention × redundancy suppression), R-KV selects tokens that are spread across the whole output, preserving critical context cues.

• Higher Information Diversity
  Unlike SnapKV—which prefers tokens near the query and often selects the same local tokens repeatedly—R-KV captures valuable pieces of information at diverse positions.

• Significantly Less Redundancy
  SnapKV still picks distant, low-value segments (e.g., “3 students are leaving early.”, “But in the initial”).
  R-KV’s redundancy check filters most of these out.

By combining attention strength with redundancy filtering, R-KV retains the important context and removes noise, successfully completing the task.
In this example, the pure attention strategy of SnapKV fails due to limited coverage and excess redundancy.

We implement visualization functions to help illustrate the multi-step token eviction pattern.

Run analysis_scripts/analysis.ipynb to see which tokens are kept at each compression step.

If you find R-KV useful in your research, please cite us:

@article{cai2025r,
  title={R-KV: Redundancy-aware KV Cache Compression for Training-Free Reasoning Models Acceleration},
  author={Cai, Zefan and Xiao, Wen and Sun, Hanshi and Luo, Cheng and Zhang, Yikai and Wan, Ke and Li, Yucheng and Zhou, Yeyang and Chang, Li-Wen and Gu, Jiuxiang and others},
  journal={arXiv preprint arXiv:2505.24133},
  year={2025}
}