Reproduce “Massive Memorization with Hundreds of Trillions of Parameters for Sequential Transducer Generative Recommenders” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhimin Chen, Chenyu Zhao, Ka Chun Mo, Yunjiang Jiang, Jane H. Lee, Khushhall Chandra Mahajan, Ning Jiang, Kai Ren and 2 more
The VISTA framework from Meta and Yale University introduces a two-stage sequential transducer generative recommender that effectively utilizes ultra-long user interaction histories (up to one million items). It achieves a 94% reduction in inference GPU resource usage and boosts online engagement metrics by decoupling history summarization from real-time inference through cached summary embeddings and a novel quasi-linear attention mechanism.
Key insights
- Decoupling the computationally intensive history summarization from real-time inference is an effective strategy for overcoming scalability challenges associated with ultra-long user histories in industrial recommendation systems.
- A specially designed quasi-linear attention mechanism can provide efficient linear-time complexity while retaining sufficient expressive power for comprehensive user preference modeling in recommendation contexts.
- Incorporating a generative sequence reconstruction loss during summarization training enhances the memorization capacity of summary embeddings, allowing them to retain critical information from the full user interaction history.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.