Reproduce “Classifying and Addressing the Diversity of Errors in Retrieval-Augmented Generation Systems” on OpenResearch

About this paper

Kin Kwan Leung, Mouloud Belbahri, Yi Sui, Alex Labach, Xueying Zhang, Stephen Anthony Rose, Jesse C. Cresswell

Researchers at Layer 6 AI created a taxonomy of 16 error types for multi-stage Retrieval-Augmented Generation (RAG) systems and an auto-evaluation tool, RAGEC, to automatically classify them. Their analysis of realistic RAG pipelines showed that chunking and retrieval errors are more common than fabricated content, and using RAGEC enabled a 10% reduction in overall error rate for a reference system.

Key insights

• Fabricated content (hallucination) is a relatively less frequent error type in realistic RAG systems compared to issues originating from upstream stages like chunking and retrieval.
• Error distributions in RAG systems are highly sensitive to dataset characteristics and document structure, demonstrating varied failure modes across different benchmarks.
• Errors frequently co-occur and propagate through the RAG pipeline, underscoring the interconnectedness and cascading nature of issues from early stages to generation.

Summary by alphaXiv.