Reproduce “Scalable Cross-Entropy Loss for Sequential Recommendations with Large Item Catalogs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Gleb Mezentsev, Danil Gusak, Ivan Oseledets, Evgeny Frolov
Researchers at Skolkovo Institute of Science and Technology developed Scalable Cross-Entropy (SCE) loss, a method to efficiently train sequential recommender systems with extensive item catalogs. This approach significantly reduces GPU memory usage and training time while maintaining or improving recommendation quality compared to existing loss approximation techniques.
Key insights
- SCE loss offers universal memory efficiency, outperforming both full CE (for large catalogs) and negative sampling methods (even for small catalogs due to their overhead).
- The 'Mix' operation substantially improves the ability to uniquely select model outputs for buckets, contributing to more stable training and notable gains in model quality.
- By selectively focusing on informative examples in a randomized, GPU-friendly manner, SCE can sometimes achieve better recommendation quality than full CE, potentially due to regularization effects or improved gradient signal focusing.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.