Reproduce “One Embedder, Any Task: Instruction-Finetuned Text Embeddings” on OpenResearch

About this paper

Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith and 2 more

INSTRUCTOR proposes a single, instruction-finetuned text embedding model that generates task- and domain-aware embeddings, achieving state-of-the-art performance with an average 3.4% improvement across 70 diverse evaluation tasks. This approach reduces the need for task-specific finetuning by enabling a universal embedder capable of adapting to varied end goals and unseen domains.

Key insights

• Natural language instructions enable text embedding models to dynamically adjust for different use cases, allowing a single model to generate task- and domain-aware embeddings.
• Instructions serve as a powerful mechanism to reconcile diverse training signals, allowing the model to benefit from a wide variety of symmetric and asymmetric datasets without performance degradation.
• Including diverse instruction formats during training improves the model's robustness to paraphrased instructions and enhances generalization to unseen domains.

Summary by alphaXiv.