Reproduce “RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment” on OpenResearch

About this paper

Zhuoran Jin, Hongbang Yuan, Tianyi Men, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao

Researchers from the Chinese Academy of Sciences introduced RAG-RewardBench, the first benchmark for evaluating reward models in Retrieval Augmented Generation (RAG) settings. This benchmark reveals that existing reward models struggle with RAG-specific preferences, achieving at most 78.3% accuracy, and current RAG-LMs show only marginal preference alignment.

Key insights

• Current general-purpose reward models are not adequately equipped to handle the nuanced, RAG-specific preference alignment demands, showing a significant performance drop in specialized scenarios.
• The benchmark highlighted that state-of-the-art SFT-trained RALMs exhibit negligible improvement (average 0.6%) in preference alignment, underscoring the urgent need for explicit preference-aligned RAG training paradigms.
• RAG-RewardBench's evaluation correlates strongly with downstream RAG task performance, validating its utility as a reliable proxy for assessing an RM's effectiveness in practical RAG applications.

Summary by alphaXiv.