Reproduce “Retro*: Optimizing LLMs for Reasoning-Intensive Document Retrieval” on OpenResearch

About this paper

Junwei Lan, Jianlyu Chen, Zheng Liu, Chaofan Li, Siqi Bao, Defu Lian

An LLM-based retrieval model, RETRO*, enables effective document retrieval for tasks requiring fine-grained reasoning by integrating a rubric-based scoring system and test-time scaling. This approach achieved an average nDCG@10 of 36.6 on the BRIGHT benchmark with its 7B model and demonstrated improved scalability and interpretable relevance assessments.

Key insights

• Rubric-based prompting allows LLMs to perform fine-grained, interpretable reasoning for document relevance, yielding concrete absolute scores rather than just relative rankings.
• Aggregating relevance scores from multiple reasoning trajectories during inference significantly enhances retrieval accuracy and robustness, enabling smaller models with scaling to outperform larger, unscaled models.
• A combined SFT and RL training approach with composite rewards effectively optimizes LLMs for the dual objectives of precise absolute relevance scoring and accurate relative document ranking simultaneously.

Summary by alphaXiv.