Reproduce “How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrieval” on OpenResearch

About this paper

Sheng-Chieh Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, Xilun Chen

The DRAGON framework introduces a BERT-base-sized dense retriever capable of achieving state-of-the-art performance in both supervised and zero-shot retrieval tasks without increasing model size. It employs a progressive, diverse data augmentation strategy combining efficient query generation with multi-teacher relevance label supervision.

Key insights

• BERT-base-sized dense retrievers' generalization limits are primarily due to sub-optimal training methodologies, not inherent model capacity, challenging a common hypothesis.
• Diverse relevance signals from multiple types of teacher retrievers are crucial for dense retriever generalization, outperforming distillation from a single strong teacher like a cross-encoder.
• Economical query augmentation methods, such as sentence cropping, contribute substantially to a dense retriever's generalization capability, sometimes more than expensive generative queries.
• The specific progression (trajectory) of introducing diverse teachers through a curriculum learning approach significantly influences zero-shot performance.

Summary by alphaXiv.