Reproduce “Ranked List Truncation for Large Language Model-based Re-Ranking” on OpenResearch

About this paper

Chuan Meng, Negar Arabzadeh, Arian Askari, Mohammad Aliannejadi, Maarten de Rijke

This paper empirically investigates the performance of various Ranked List Truncation (RLT) methods within the retrieve-then-re-rank paradigm, specifically when using Large Language Model (LLM) based re-rankers. It demonstrates that simple RLT approaches, such as shallow fixed cut-offs, can be highly effective in reducing computational costs while maintaining effectiveness, particularly when paired with high-quality first-stage retrievers.

Key insights

• Findings from RLT applied to single-stage retrieval do not consistently generalize to the retrieve-then-re-rank paradigm, particularly with LLM re-rankers.
• Supervised RLT methods, despite their complexity, generally do not surpass simpler unsupervised methods or well-tuned fixed cut-offs in the re-ranking context.
• The choice of the first-stage retriever significantly impacts RLT performance and optimal truncation depth, whereas the type of re-ranker (LLM vs. PLM) has limited influence.

Summary by alphaXiv.