Reproduce “SGPT: GPT Sentence Embeddings for Semantic Search” on OpenResearch

About this paper

Niklas Muennighoff

This research introduces SGPT (Sentence GPT), a framework for adapting large decoder-only transformer models, traditionally used for generation, to semantic search and sentence embedding tasks. The work demonstrates that both a Cross-Encoder (SGPT-CE) and a Bi-Encoder (SGPT-BE) variant can achieve new state-of-the-art performance on benchmarks like BEIR, with SGPT-BE-5.8B improving nDCG@10 by 7% over previous Bi-Encoder models.

Key insights

• Large decoder-only models, when properly adapted, can achieve superior performance in semantic search and sentence embedding, challenging the traditional reliance on encoder-only architectures.
• A novel position-weighted mean pooling strategy effectively compensates for the causal attention mechanism of decoders, improving the quality of extracted embeddings.
• Parameter-efficient fine-tuning methods like BitFit enable the adaptation of multi-billion parameter decoder models with minimal computational overhead, maintaining competitive performance.

Summary by alphaXiv.