Reproduce “Teaching Dense Retrieval Models to Specialize with Listwise Distillation and LLM Data Augmentation” on OpenResearch

About this paper

Manveer Singh Tamber, Suleman Kazi, Vivek Sourabh, Jimmy Lin

Researchers from the University of Waterloo and Vectara developed a fine-tuning strategy for dense retrieval models, employing listwise distillation from a RankT5-3B cross-encoder and diverse query generation via Llama-3.1. This approach consistently enhanced model specialization, with BGE-base seeing NDCG@10 improvements (e.g., 70.2 to 71.8 on DL19), and demonstrated that LLM-generated synthetic queries can rival human-written data for training.

Key insights

• Fine-tuning state-of-the-art dense retrieval models with standard InfoNCE contrastive loss often degrades their effectiveness, while listwise distillation consistently yields improvements.
• LLM-generated synthetic queries, especially when diverse, prove to be highly competitive with human-written queries for training retrieval models.
• The effectiveness of the cross-encoder teacher, such as RankT5-3B, acts as a primary limiting factor for achieving even greater specialization gains in dense retrieval.

Summary by alphaXiv.