Reproduce “Efficiency Unleashed: Inference Acceleration for LLM-based Recommender Systems with Speculative Decoding” on OpenResearch

About this paper

Yunjia Xi, Hangyu Wang, Bo Chen, Jianghao Lin, Menghui Zhu, Weiwen Liu, Ruiming Tang, Zhewei Wei and 2 more

Researchers from Shanghai Jiao Tong University and Huawei Noah’s Ark Lab introduced LASER, a framework optimizing speculative decoding for accelerating Large Language Model inference in recommender systems' offline knowledge generation. This method achieved a 3-5x speedup on public datasets and saved 67% of computational resources in an online commercial advertising scenario, all while maintaining lossless performance in downstream recommendation tasks.

Key insights

• Even for offline knowledge generation, LLM inference efficiency remains a critical bottleneck for industrial-scale recommender systems, demanding optimized acceleration techniques.
• Retrieval-based speculative decoding is particularly suitable for recommender system knowledge generation given the continuous need for new knowledge and the reusability of existing information.
• Downstream recommendation tasks exhibit high diversity tolerance for LLM-generated texts, allowing for a relaxation of speculative decoding's verification criteria to boost acceleration without compromising recommendation quality.

Summary by alphaXiv.