Reproduce “DREAM: Dense Retrieval Embeddings via Autoregressive Modeling” on OpenResearch

About this paper

Yixuan Tang, Yi Yang

Researchers from The Hong Kong University of Science and Technology developed DREAM, a method for training dense retrieval embeddings by leveraging a frozen Large Language Model's next-token prediction objective. This approach integrates retriever scores into specific LLM attention heads and significantly surpasses traditional contrastive learning and other LLM-supervised baselines on multiple retrieval benchmarks, reducing reliance on explicit relevance labels.

Key insights

• The autoregressive next-token prediction objective of a frozen LLM can provide a robust and effective supervisory signal for training standalone dense retrievers, eliminating the need for explicit positive/negative labels.
• Injecting retriever scores into specific 'query-focused retrieval heads' within the LLM is critical for maximizing performance, validating prior work on attention head specialization and demonstrating its practical application.
• The implicit competition among candidate documents during the NTP objective, due to score normalization, helps the retriever learn a less collapsed embedding space with improved uniformity.

Summary by alphaXiv.