Reproduce “xGR: Efficient Generative Recommendation Serving at Scale” on OpenResearch

About this paper

Qingxiao Sun, Tongxuan Liu, Shen Zhang, Siyu Wu, Peijun Yang, Haotian Liang, Menxin Li, Xiaolong Ma and 8 more

A research collaboration from Beihang University, JD Company, University of Science and Technology Beijing, and Huawei developed xGR, an efficient serving system specifically for generative recommendation models. xGR optimizes attention computation, beam search, and system scheduling, achieving up to 8.05x higher throughput for GR models and significantly reducing memory consumption by over 75% compared to existing LLM serving frameworks under stringent low-latency conditions.

Key insights

• Generative recommendation workloads pose distinct challenges to general LLM serving systems, primarily due to long input sequences, short outputs, and the computational intensity of beam search with large beam widths.
• Decoupling KV cache management for shared prompt prefixes and beam-specific tokens significantly reduces memory overhead and improves memory bandwidth utilization.
• Algorithmic enhancements to beam search, such as valid path constraints and early sorting termination, are crucial for managing the massive candidate space and maintaining efficiency.

Summary by alphaXiv.