Reproduce “CSRv2: Unlocking Ultra-Sparse Embeddings” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Lixuan Guo, Yifei Wang, Tiansheng Wen, Yifan Wang, Aosong Feng, Bo Chen, Stefanie Jegelka, Chenyu You

CSRv2 introduces a training methodology that enables the practical use of ultra-sparse embeddings, achieving up to 300x compute and memory efficiency and a 7x retrieval speedup while mitigating previous performance degradation. The method reduces inactive neurons at k=2 from over 80% to around 20%, leading to accuracy improvements of up to 14% over prior approaches on various text tasks.

Key insights

  • Ultra-sparse embeddings are shown to be viable for practical deployment, challenging the assumption that their representational capacity is inherently limited.
  • A curriculum learning strategy (k-annealing) combined with supervised contrastive signals can effectively mitigate the "massive dead neuron" problem, significantly increasing the effective utilization of latent dimensions.
  • Aligning sparse representations with supervised, task-relevant semantics through contrastive learning is crucial for preserving and enhancing performance under extreme sparsity.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.