Reproduce “Jasper and Stella: distillation of SOTA embedding models” on OpenResearch

About this paper

Dun Zhang, Jiacheng Li, Ziyang Zeng, Fulong Wang

The Jasper model achieves an MTEB score of 71.54, positioning it as the No.3 ranked text embedding model with 2 billion parameters, a performance comparable to models 3.5 times larger. This efficiency is enabled by a multi-stage knowledge distillation framework employing three distinct loss functions, Matryoshka Representation Learning, and multimodal alignment.

Key insights

• A multi-stage knowledge distillation framework incorporating specialized loss functions can effectively transfer complex semantic understanding from large teacher models to significantly smaller student embedding models.
• Achieving near state-of-the-art embedding performance is possible with a model (2 billion parameters) substantially smaller than top-ranked counterparts (7 billion parameters), bridging the gap between accuracy and computational efficiency.
• Matryoshka Representation Learning (MRL) can be successfully integrated into the distillation process to enable efficient dimension reduction of embeddings without major performance degradation, enhancing practical deployability.

Summary by alphaXiv.