Reproduce “The Distracting Effect: Understanding Irrelevant Passages in RAG” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chen Amiraz, Florin Cuconasu, Simone Filice, Zohar Karnin
Researchers at the Technology Innovation Institute (TII) and Sapienza University of Rome formally define and quantify the 'distracting effect' of irrelevant passages on LLM performance in Retrieval Augmented Generation (RAG) systems. They show that explicitly training LLMs with these hard distracting passages improves RAG accuracy by up to 7.5% and enhances the models' ability to abstain from incorrect answers.
Key insights
- The "distracting effect" is a quantifiable measure of an irrelevant passage's interference with an LLM's ability to correctly respond to a query.
- Hard distracting passages, which are semantically related to a query but do not contain the answer, pose a substantial risk to RAG performance.
- The distracting effect is consistent across various LLMs, indicating it is an intrinsic property of the passage rather than model-specific behavior.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.