Reproduce “Redefining Retrieval Evaluation in the Era of LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Giovanni Trappolini, Florin Cuconasu, Simone Filice, Yoelle Maarek, Fabrizio Silvestri
Researchers from Sapienza University of Rome and TII introduced an evaluation framework for Retrieval Augmented Generation (RAG) systems, comprising a passage utility annotation schema and Utility and Distraction-aware Cumulative Gain (UDCG) metrics. This framework addresses the limitations of traditional IR metrics by capturing LLM-specific document processing behaviors, including the negative impact of distractors, and demonstrates up to 36% higher correlation with end-to-end RAG answer accuracy compared to nDCG. This enables more precise optimization of RAG retrieval components.
Key insights
- Irrelevant 'distractor' passages can substantially reduce LLM answer accuracy, with a single relevant passage performing up to 9 accuracy points worse when surrounded by hard distractors compared to weak ones.
- LLMs exhibit U-shaped positional biases (e.g., 'lost-in-the-middle' effect) that conflict with the monotonically decreasing positional discounts found in traditional IR metrics.
- Both the continuous nature of passage utility (distinguishing between beneficial and detrimental effects) and the explicit accounting for negative distracting effects are crucial for accurate RAG retrieval evaluation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.