Reproduce “SMEC: Rethinking Matryoshka Representation Learning for Retrieval Embedding Compression” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Biao Zhang, Lixin Chen, Tong Liu, Bo Zheng
The SMEC framework optimizes embedding compression for high-dimensional Large Language Model outputs in retrieval systems by addressing gradient instability and information loss. It achieves superior performance and up to 14x lossless compression on text, image, and multimodal datasets, improving scalability and efficiency for industrial applications.
Key insights
- Sequencing the training of different embedding dimensions significantly reduces gradient variance, leading to faster convergence and improved performance compared to parallel training.
- Dynamically learning which dimensions are most important (via ADS) for lower-dimensional representations preserves more critical semantic information than fixed truncation rules.
- Leveraging a broader, carefully selected memory of hard samples (via S-XBM) for unsupervised contrastive learning is crucial for maintaining semantic integrity during aggressive compression.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.