Reproduce “Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Serving” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xiaoxiang Shi, Colin Cai, Junjia Du, Zhihao Jia

Nexus presents an intra-GPU logical disaggregation framework for serving large language models, dynamically partitioning Streaming Multiprocessors between prefill and decode phases on a single GPU. The system delivers 1.5–1.9x higher throughput and 2–20x lower Time-to-First-Token compared to monolithic systems, often matching multi-GPU disaggregated solutions with half the hardware.

Key insights

  • Empirical analysis revealed that mixed prefill-decode batches in monolithic systems cause 8-10x higher decode latency due to significant phase interference.
  • Both prefill and decode phases exhibit diminishing returns with increased GPU compute allocation; decode latency saturates quickly beyond 50% SMs, indicating that full GPU allocation to a single phase is inefficient.
  • Dynamic memory contention significantly impacts decode performance, with decode latency increasing by 36% when prefill KV cache length grows, necessitating real-time resource reallocation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.