Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

UCCL is an efficient communication library for GPUs, covering collectives, P2P (e.g., KV cache transfer, RL weight transfer), and EP (e.g., GPU-driven)

[Neurips 2025] R-KV: Redundancy-aware KV Cache Compression for Reasoning Models

Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond

Inference-native Tokenmaxxing Agent Harness for Loop Engineering

Persist and reuse KV Cache to speedup your LLM.

Alibaba Cloud's high-performance KVCache system for LLM inference, with components for global cache management, inference simulation(HiSim), and more.

PiKV: KV Cache Management System for Mixture of Experts [Efficient ML System]

Sardeenz is a proof-of-concept application that allows you to load more than one model on a given GPU. It allows you to add more and more models onto a GPU, until it is fully utilized.

CacheRoute is an innovative LLM scheduling scheme dedicated to enabling flexible KV cache reuse across LLM systems, improving task performance and system efficiency.

(ACL2025 oral) SCOPE: Optimizing KV Cache Compression in Long-context Generation

🔥 [ICML'26] ParisKV: Fast and Drift-Robust KV-Cache Retrieval for Long-Context LLMs

Official implementation of "Shadow in the Cache: Unveiling and Mitigating Privacy Risks of KV-cache in LLM Inference" (NDSS 2026)

Span Queries: What if we had a way to plan and optimize GenAI like we do for SQL?

A TurboQuant implementation with Llama.cpp for AMD with Vulkan runtime

inference cookbook / inference 框架原理解析

High-Performance KV Cache Sharing Library

An empirical study of benchmarking LLM inference with KV cache offloading using vLLM and LMCache on NVIDIA GB200 with high-bandwidth NVLink-C2C .

KV Cache with PagedAttention vs PagedAttention + TurboQuant - experiments across token sizes comparing memory, latency, and accuracy.

KV-cache compression for LLMs: reference implementations of TurboAngle and TurboQuant codecs with Triton GPU kernels