Reproduce “Are LLM-based Recommenders Already the Best? Simple Scaled Cross-entropy Unleashes the Potential of Traditional Sequential Recommenders” on OpenResearch

About this paper

Cong Xu, Zhangchi Zhu, Mo Yu, Jun Wang, Jianyong Wang, Wei Zhang

Researchers demonstrated that traditional sequential recommenders, when optimized with a full softmax cross-entropy (CE) loss, consistently outperform fine-tuned LLM-based recommenders in terms of ranking accuracy. They also introduced Scaled Cross-Entropy (SCE), an efficient approximation of CE loss, enabling traditional models to maintain this strong performance in large-scale practical settings.

Key insights

• The perceived superior performance of LLM-based recommenders stemmed significantly from inconsistent experimental setups and the choice of loss function, rather than inherent architectural advantages.
• Optimal recommendation performance with CE loss arises from a balanced trade-off between "tightness" (how accurately the loss reflects ranking metrics) and "coverage" (the breadth of negative items considered).
• Scaled Cross-Entropy (SCE) effectively addresses the scalability issue of full softmax CE, allowing traditional models to leverage its benefits with a small number of negative samples without sacrificing ranking quality.

Summary by alphaXiv.