Reproduce “Rank4Gen: RAG-Preference-Aligned Document Set Selection and Ranking” on OpenResearch

About this paper

Yongqi Fan, Yuxiang Chu, Zhentao Xia, Xiaoyang Chen, Jie Liu, Haijin Liang, Jin Ma, Ben He and 4 more

Researchers from East China University of Science and Technology and Tencent developed Rank4Gen, a novel ranking model for Retrieval-Augmented Generation (RAG) systems that performs generator-aware document set selection and ordering. The study also created PRISM, a bilingual preference-aligned dataset, demonstrating that Rank4Gen improved downstream response quality with F1 gains of +1.07 to +2.08 over strong baselines and generalized to unseen large language models.

Key insights

• A controlled study empirically showed that both the composition of document subsets and their ordering critically impact downstream RAG response quality, even among individually relevant documents.
• Optimal document selection and ordering preferences are not universal but vary systematically across different large language models.
• Optimizing ranking directly for downstream answer quality through preference learning, rather than solely relevance, leads to more stable and performant RAG systems.

Summary by alphaXiv.