Reproduce “Long-Sequence Recommendation Models Need Decoupled Embeddings” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ningya Feng, Junwei Pan, Jialong Wu, Baixu Chen, Ximei Wang, Qian Li, Xian Hu, Jie Jiang and 1 more

Researchers from Tsinghua University and Tencent Inc. introduce Decoupled Attention and Representation Embeddings (DARE) for long-sequence recommendation models, resolving gradient interference between attention and representation learning. DARE yields up to a 9‰ AUC increase offline and a 1.47% Gross Merchandise Value lift in online A/B tests on a large advertising platform, while also accelerating inference.

Key insights

  • Attention and representation learning are distinct tasks with conflicting gradients that suffer from interference when sharing embeddings in long-sequence recommendation models.
  • Existing NLP-derived decoupling techniques fail in recommendation due to specific domain constraints, such as typically smaller embedding dimensions.
  • Complete decoupling of attention and representation embeddings resolves task interference, enabling independent optimization for each component and leading to enhanced accuracy and efficiency.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.