Reproduce “FeB4RAG: Evaluating Federated Search in the Context of Retrieval Augmented Generation” on OpenResearch

About this paper

Shuai Wang, Ekaterina Khramtsova, Shengyao Zhuang, Guido Zuccon

FeB4RAG provides an evaluation collection for resource selection and result merging in retrieval-augmented generation (RAG). It contains 790 chatbot-style requests, 16 BEIR-based search resources, and relevance judgments derived from large language models. In an author-assessed sample of 80 requests, oracle-selected and ranked results yielded more-preferred answers than all-resource round-robin retrieval; this does not test a practical selector.

Key insights

• LLMs can reliably generate relevance judgments for test collections, achieving inter-annotator agreement (Cohen's Kappa of 0.57 between two LLMs) comparable to human assessors.
• Most user information needs benefit from federated search, with queries often having relevant content across multiple data sources (average 11.9 out of 16 simulated engines).
• Sophisticated federated search strategies, including intelligent resource selection and result merging, are crucial for high-quality Retrieval-Augmented Generation (RAG), significantly outperforming naive approaches.

Summary by alphaXiv.