Reproduce “RAGged Edges: The Double-Edged Sword of Retrieval-Augmented Chatbots” on OpenResearch

About this paper

Philip Feldman, James R. Foulds, Shimei Pan

Feldman, Foulds, and Pan from the University of Maryland, Baltimore County, empirically evaluated Retrieval-Augmented Generation (RAG) in chatbots by having human experts evaluate LLM responses concerning their own CVs. Their work reveals that while RAG dramatically increased accuracy eighteenfold to 93.95%, a distinct 6.04% of errors persisted even with perfect context, detailing specific failure modes like 'noisy context' and 'context-based synthesis'.

Key insights

• Retrieval-Augmented Generation (RAG) significantly improves LLM accuracy, yet a persistent error rate of 6.04% remains even when perfect, relevant context is provided.
• Errors with RAG are not random but fall into distinct categories, including 'noisy context' (misplaced information) and a concerning 'context-based synthesis' where LLMs fabricate plausible details by interpolating from other factual data.
• The study underscores that supplying accurate context is necessary but not sufficient for fully reliable LLM output, as the model's inherent generative capabilities can still subtly distort or invent information.

Summary by alphaXiv.