Reproduce “Investigating the Robustness of Retrieval-Augmented Generation at the Query Level” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sezen Perçin, Xin Su, Qutub Sha Syed, Phillip Howard, Aleksei Kuvshinov, Leo Schwinn, Kay-Ulrich Scholl
This research from TUM, Intel Labs, and Thoughtworks systematically investigates how retrieval-augmented generation (RAG) systems respond to various query-level perturbations, such as typos and semantic variations. The study introduces a framework to decouple the sensitivity of retrievers and generators, demonstrating module-specific robustness profiles across diverse datasets and model architectures.
Key insights
- Dense retrievers are more robust to redundant query information, while sparse retrievers show greater resilience to typographical errors.
- LLM generators exhibit varied robustness depending on the dataset and the specific model, with ambiguity consistently degrading performance in both closed-book and oracle settings.
- In general-domain contexts (NQ), end-to-end RAG robustness is largely dictated by the retriever's performance, but in domain-specific settings (BioASQ), the generator's ability to utilize context despite query perturbations becomes more critical.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.