Reproduce “Scaling Sparse and Dense Retrieval in Decoder-Only LLMs” on OpenResearch

About this paper

Hansi Zeng, Julian Killingback, Hamed Zamani

Researchers at the University of Massachusetts Amherst systematically investigated the scaling behavior of sparse and dense retrieval models within decoder-only LLMs using Llama-3. Their work found that sparse retrieval consistently outperformed dense methods across benchmarks and that a combined contrastive loss and knowledge distillation objective achieved state-of-the-art results, notably improving BEIR performance by 10.6% over ColBERTv2 with an 8B sparse model.

Key insights

• Scaling performance gains were evident with Contrastive Loss, but Knowledge Distillation showed minimal or negative scaling benefits, especially on out-of-domain tasks, suggesting teacher models can limit larger student LLMs.
• Sparse retrieval consistently outperformed dense retrieval across all benchmarks and fine-tuning objectives, demonstrating superior robustness and generalization in zero-shot settings.
• Combining Contrastive Loss and Knowledge Distillation proved most effective, leveraging the strengths of both to achieve state-of-the-art performance while maintaining robustness.

Summary by alphaXiv.