PDD: Unleashing Economical and Flexible Heterogeneous LLM Inference via Cross-Datacenter Prefill-Decode Disaggregation

Infinigence-AI  |  📄 Technical Report

• [2026-07] The PDD technical report is now publicly released. Future progress and public materials of this project will be updated in this repository. Stay tuned!

Interconnecting geographically dispersed clusters over low-cost, wide-area Ethernet is a scalable and cost-effective alternative to building dedicated intra-datacenter heterogeneous clusters for LLM inference. However, this disaggregated architecture imposes heavy KV-cache communication between clusters, causing substantial Time-to-First-Token (TTFT) latency — especially detrimental for agentic workloads, which are characterized by long contexts, high prefix-cache hit rates, and short output lengths.

PDD is a three-tier disaggregation architecture built on top of Prefill-Decode (PD) disaggregation, consisting of:

• Prefill (P) Instance — handles the compute-bound prefill phase on compute-heavy hardware in the primary datacenter.
• RelayDecode (RLD) Instance — co-located with P in the same datacenter over high-speed RDMA. It receives the KV cache within tens of milliseconds after prefill and eagerly starts decoding, masking the much slower cross-datacenter KV transfer.
• MainDecode (MD) Instance — deployed in a geographically distant datacenter over low-cost wide-area Ethernet (typically < 100 Gbps). It receives the KV cache via TCP, seamlessly takes over decoding from RLD, and completes the bulk of token generation.

• Decode-side Radix Cache (DRC). Caching historical KV cache on decode instances reduces required cross-datacenter bandwidth by up to 10× at a 90% average prefix hit rate, mitigating the wide-area bandwidth bottleneck.
• Extend-Decode Handoff. Instead of transmitting the bulky incremental KV cache from RLD to MD, RLD only sends lightweight CPU-resident Token IDs; MD locally recomputes the incremental KV cache while generating the next token. Two modes are supported and can be alternated dynamically based on RLD load:
  • Catch-Up Handoff — MD gradually catches up to RLD's progress via step-by-step Extend-Decode; fully transparent to users with no OTPS fluctuation.
  • One-Shot Handoff — RLD transfers all eager tokens at once and immediately frees resources, minimizing RLD occupation at the cost of a brief OTPS dip.
• Multi-Stage Pipeline Orchestration. Requests are routed by cache hit rate: the vast majority (> 70%) of high-hit-rate requests go directly through the P-MD pipeline, while low-hit-rate outliers go through P-RLD-MD for latency masking. This bounds RLD occupation time and keeps the MD-side DRC hit rate on par with the P-side prefix cache.
• Fine-Grained Heterogeneous Deployment. Since RLD only processes a small fraction of initial tokens for low-hit-rate requests, providers need only a minimal fraction of memory-access-efficient chips inside a compute-heavy datacenter, offloading the bulk of memory-bound decoding to remote MD clusters — minimizing heterogeneous hardware cost and resource waste under shifting workloads.

Evaluated on DeepSeek-V4-pro with real-world agentic workload traces (64K average context, 90% average prefix hit rate with a highly skewed distribution), on a testbed of 43×8 H100 GPUs (P + RLD) and 32×8 H200 GPUs (MD) interconnected via 2×10 Gbps wide-area Ethernet:

PDD is orthogonal and complementary to scheduling-centric cross-datacenter designs such as PrfaaS (Prefill-as-a-Service): PrfaaS optimizes SLA fairness via request scheduling, while PDD optimizes end-to-end SLOs via latency masking — the two can be naturally integrated.

This repository currently hosts the technical report. Future progress of the project and related public materials will be updated here.

If you find PDD useful in your research or production systems, please cite:

@techreport{wang2026pdd,
  title        = {PDD: Unleashing Economical and Flexible Heterogeneous LLM Inference via Cross-Datacenter Prefill-Decode Disaggregation},
  author       = {Wang, Yida and Li, Xiuhong and Ma, Jianping and Sun, Gan and Xu, Yunshen and Han, Buhe and Ng, Jingxu and Luo, Yuhao and Hong, Ke and Dai, Guohao and Li, Boxun and Wang, Yu},
  institution  = {Infinigence-AI},
  year         = {2026}
}

Yida Wang¹, Xiuhong Li¹, Jianping Ma¹, Gan Sun¹, Yunshen Xu¹, Buhe Han¹, Jingxu Ng¹˒⁴, Yuhao Luo¹˒⁴, Ke Hong¹˒², Guohao Dai¹˒³*, Boxun Li¹*, Yu Wang²*

¹ Infinigence-AI   ² Tsinghua University   ³ Shanghai Jiao Tong University   ⁴ Peking University   (* corresponding authors)

The cross-datacenter transfer stack of PDD builds upon the TCP transfer engine of Mooncake (KVCache.AI). We also thank the open-source LLM serving community, including SGLang, for the foundations this work stands on.

For questions and collaborations, please open an issue in this repository or contact Boxun Li (liboxun@infini-ai.com).