Reproduce “Retrieval-Augmented Generation with Conflicting Evidence” on OpenResearch

About this paper

Han Wang, Archiki Prasad, Elias Stengel-Eskin, Mohit Bansal

Researchers at UNC Chapel Hill introduced MADAM-RAG, a multi-agent framework, and RAMDocs, a dataset, to improve Retrieval-Augmented Generation (RAG) systems' capacity to navigate complex information conflicts including ambiguity, misinformation, and noise. MADAM-RAG achieved up to an 11.40% accuracy improvement on ambiguous queries and demonstrated superior robustness against misinformation and imbalanced evidence compared to existing baselines.

Key insights

• A multi-agent deliberative approach, where agents individually process information and then engage in a debate, is more effective at handling intertwined information conflicts than single-agent or simple document concatenation methods.
• The aggregator module plays a crucial role in synthesizing coherent answers and accurately discerning between ambiguous query situations that require multiple valid answers and misinformation that must be suppressed.
• Iterative debate rounds enable agents to reduce errors and improve their responses by collectively reinforcing valid evidence while challenging and revising claims based on conflicting information.

Summary by alphaXiv.