Reproduce “Efficient Listwise Reranking with Compressed Document Representations” on OpenResearch

About this paper

Hervé Déjean, Stéphane Clinchant

Researchers at NAVER LABS Europe developed RRK, an efficient listwise reranker that leverages rich, learned compressed document representations to overcome the computational expense of LLM-based systems. This framework, based on an 8B-parameter Qwen model, achieved up to 17x faster inference compared to existing 4B parameter rerankers while maintaining strong ranking effectiveness, particularly excelling with long documents.

Key insights

• Jointly training an LLM-based compressor with a listwise reranker enables the generation of task-adaptive compressed representations that preserve sufficient semantic information for high reranking effectiveness.
• Leveraging 'soft, task-adaptive compression' derived directly from an LLM's internal representations significantly outperforms using frozen, pre-trained compressors or fixed IR embeddings, highlighting the importance of fine-grained, learned representations for downstream tasks.
• Drastically reducing the effective input sequence length through rich document compression alleviates the quadratic attention cost, allowing larger 8B-parameter LLMs to achieve superior efficiency (up to 17x faster) while maintaining or improving performance over smaller models.

Summary by alphaXiv.