Reproduce “Quantized Inference for OneRec-V2” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yi Su, Xinchen Luo, Hongtao Cheng, Ziteng Shu, Yunfeng Zhao, Fangyu Zhang, Jiaqiang Liu, Xiao Liang and 2 more

Researchers at Kuaishou Inc. successfully applied FP8 quantized inference to the OneRec-V2 generative recommendation model, achieving a 49% reduction in inference latency and a 92% increase in throughput in a production environment. This optimization was realized without any degradation in core online recommendation quality metrics, demonstrating the efficiency gains possible for modern generative recommender systems.

Key insights

  • OneRec-V2, a generative recommendation model, exhibits numerical distribution characteristics for weights and activations that are significantly closer to those of LLMs than traditional recommender models, making it more amenable to low-precision inference.
  • Translating low-precision computation into substantial end-to-end system benefits requires a comprehensive optimization strategy that combines selective quantization with a deeply optimized inference infrastructure and specialized operator kernels.
  • Modern generative recommendation models can leverage techniques developed for LLMs to achieve significant inference efficiency gains (near 2x throughput) without compromising critical online recommendation quality in production settings.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.