Reproduce “Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hao-Jun Michael Shi, Dheevatsa Mudigere, Maxim Naumov, Jiyan Yang
A framework for compositional embeddings using complementary partitions enables memory-efficient deep learning recommendation systems. This method generates unique representations for high-cardinality categorical features by combining smaller embedding tables, significantly reducing memory footprint while maintaining model quality superior to the traditional hashing trick.
Key insights
- Complementary partitions provide a principled way to create memory-efficient yet unique representations for high-cardinality categories, addressing the core limitation of the hashing trick.
- Element-wise multiplication proved to be the most scalable and effective operation for composing embeddings, offering a strong trade-off between model compression and preserved quality.
- The framework is flexible, allowing for partitions derived from mathematical properties (like quotient-remainder or Chinese Remainder Theorem) or potentially inherent data characteristics.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.