Reproduce “Uncertainty Quantification in Retrieval Augmented Question Answering” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Laura Perez-Beltrachini, Mirella Lapata
Researchers at the University of Edinburgh developed a method for quantifying uncertainty in Retrieval Augmented Question Answering (RAG) models by estimating the utility of individual retrieved passages. This approach utilizes a lightweight neural network to predict passage utility, leading to competitive performance in detecting incorrect answers (e.g., an average AUROC of 0.79 on Gemma2-9B) while offering substantial improvements in computational efficiency.
Key insights
- Quantifying the utility of retrieved passages for a specific QA model is an effective and efficient proxy for estimating the overall uncertainty of the RAG system's answer.
- A lightweight, specifically trained predictor can achieve comparable or better uncertainty quantification performance than computationally intensive sampling-based or LLM self-assessment methods.
- The method's ability to identify less useful passages can improve RAG reliability through selective answering and also serve as an effective mechanism for reranking retrieved evidence.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.