Reproduce “MERGE: Next-Generation Item Indexing Paradigm for Large-Scale Streaming Recommendation” on OpenResearch

About this paper

Jing Yan, Yimeng Bai, Zongyu Liu, Yahui Liu, Junwei Wang, Jingze Huang, Haoda Li, Sihao Ding and 2 more

The MERGE framework introduces a new item indexing paradigm for large-scale streaming recommendation systems, overcoming limitations of Vector Quantization (VQ) methods by dynamically creating and managing item clusters. It significantly improves item-to-cluster assignment accuracy, cluster occupancy uniformity, and inter-cluster separation, leading to enhanced user engagement and resource efficiency in industrial deployments.

Key insights

• Dynamically generating new clusters and adaptively updating existing ones based on item similarity significantly improves the accuracy and distinctiveness of item representations compared to static clustering approaches.
• Explicitly monitoring and managing cluster occupancy in real-time is crucial for maintaining balanced cluster sizes and mitigating popularity bias towards long-tail items.
• A hierarchical approach to item indexing, combined with dynamic cluster management, provides a more robust and scalable solution for the rapidly evolving and large-scale data in streaming recommendation environments.

Summary by alphaXiv.