Reproduce “Detecting and Repairing Hallucinations in Retrieval-Augmented Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sai Krishna Reddy Mulakkayala, Niki van Stein, Aske Plaat
A study from LIACS, Leiden University, systematically evaluates three claim-level repair strategies for hallucinations in Retrieval-Augmented Generation (RAG) systems. The research identifies a consistent trade-off between reducing judged hallucinations and retaining original content, while also assessing the reliability of detector scaling and automatic LLM-based evaluation.
Key insights
- Increasing detector model size from 7B to 13B parameters did not reliably yield a statistically significant performance improvement, indicating variability in parameter-efficient fine-tuning.
- A consistent "grounding-preservation trade-off" exists, where strategies achieving greater hallucination reduction also retain less original content.
- LLM-based automatic judges, while agreeing on the relative effectiveness of repair strategies, exhibit low inter-judge agreement on absolute hallucination rates and weak concordance with human annotations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.