Reproduce “Training Sparse Mixture Of Experts Text Embedding Models” on OpenResearch

About this paper

Zach Nussbaum, Brandon Duderstadt

Nomic AI developed Nomic Embed v2, the first general-purpose Mixture of Experts (MoE) text embedding model, to address efficiency challenges in high-performance retrieval systems. The model achieves competitive retrieval performance on both monolingual and multilingual benchmarks with significantly fewer active parameters during inference compared to traditional dense models, leveraging a modified XLM-Roberta base and large-scale contrastive pretraining with hard negative mining.

Key insights

• Integrating Mixture of Experts (MoE) into text embedding models allows for competitive retrieval performance with significantly fewer active parameters during inference compared to dense models.
• Selective conversion of MLP layers to MoE (e.g., 6 out of 12) can be more effective than converting all layers for optimal performance.
• Large-scale, high-quality data, curated via consistency filtering and positive-aware hard negative mining, is crucial for MoE embedding models to achieve their full potential, particularly in multilingual contexts.

Summary by alphaXiv.