Reproduce “Conventional Contrastive Learning Often Falls Short: Improving Dense Retrieval with Cross-Encoder Listwise Distillation and Synthetic Data” on OpenResearch

About this paper

Manveer Singh Tamber, Suleman Kazi, Vivek Sourabh, Jimmy Lin

This research reveals that conventional InfoNCE contrastive learning often degrades dense retrieval effectiveness when fine-tuning strong pre-trained models. It presents a framework utilizing cross-encoder listwise distillation and diverse synthetic query generation from a lightweight LLM to improve dense retrieval, resulting in a BERT-base model, cadet-embed-base-v1, that achieves state-of-the-art performance among its class.

Key insights

• Fine-tuning strong, pre-trained dense retrievers solely with InfoNCE contrastive loss frequently degrades performance, challenging a widely adopted practice.
• Cross-encoder listwise distillation consistently and robustly improves dense retrieval effectiveness by transferring nuanced relevance judgments from powerful teacher models.
• Training with a diverse mixture of LLM-generated synthetic query types yields stronger retrieval results than single query types, demonstrating comparable utility to human-written queries.

Summary by alphaXiv.