Reproduce “Making Large Language Models Efficient Dense Retrievers” on OpenResearch

About this paper

Yibin Lei, Shwai He, Ang Li, Andrew Yates

A two-stage compression framework, EffiR, enables large language models to serve as efficient dense retrievers by reducing model parameters by 52% and increasing query-side inference speed nearly two-fold, achieving 54.3 nDCG@10. This work identifies that MLP layers, rather than attention layers, are the primary source of redundancy in LLMs adapted for retrieval tasks.

Key insights

• MLP layers are considerably more redundant and prunable for dense retrieval tasks compared to attention layers, which remain critical for semantic aggregation.
• The importance of attention layers for retrieval is heightened when using last-token embeddings, while mean-pooling offers more robustness to attention layer removal.
• A two-stage, coarse-to-fine structural pruning approach effectively optimizes LLM architecture for retrieval, achieving better efficiency-performance trade-offs than single-stage methods.

Summary by alphaXiv.