Reproduce “Reward-RAG: Enhancing RAG with Reward Driven Supervision” on OpenResearch

About this paper

Thang Nguyen, Peter Chin, Yu-Wing Tai

The Reward-RAG framework from Dartmouth College enhances Retrieval-Augmented Generation (RAG) models by aligning their retrieval with human preferences through reward-driven supervision. This method, leveraging GPT-4o for synthetic feedback, leads to improved performance on general and domain-specific QA tasks, achieving a 92.3% accuracy on FEVER and a 60.0 NDCG@10 on NQ.

Key insights

• Applying reward-driven supervision directly to the retrieval component of RAG significantly enhances the quality and relevance of retrieved documents, a critical bottleneck for overall RAG performance.
• Sophisticated LLMs like GPT-4o can serve as effective "CriticGPTs" to synthesize high-quality training data for reward models, substantially reducing the reliance on costly human annotation.
• Structured prompting methods for LLM-based evaluators, such as "think step-by-step," yield more consistent and robust relevance annotations compared to simpler in-context learning approaches.

Summary by alphaXiv.