Reproduce “What Would Fix This RAG Failure? Auditing Counterfactual Response with Paired Evidence Interventions” on OpenResearch

About this paper

Wenzhang Du

This research introduces Pair-ID, an offline audit, to systematically measure how failed Retrieval-Augmented Generation (RAG) systems respond to counterfactual evidence interventions by adding missing support and deleting nonsupport. The study quantifies the prevalence of repairable RAG failures and demonstrates that while marginal evidence sensitivities are consistent across different LLM readers, the exact counterfactual response for a specific query remains conditional on the reader and is not precisely predictable from the original failed state.

Key insights

• A measurable proportion of RAG failures exhibit specific evidence sensitivities: approximately one-third of failures are addition-sensitive and over one-tenth are deletion-sensitive, demonstrating repairability through targeted interventions.
• The observed repair effects are primarily due to the semantic content of the added support or deleted nonsupport, rather than superficial changes to prompt length or structure.
• A "counterfactual action gap" exists, as even advanced observation-only models struggle to precisely predict the exact counterfactual response vector from the original failed state.
• While marginal evidence sensitivities (e.g., overall improvement from adding support) are reproducible across different LLM readers, the precise counterfactual response for a given query is conditional on the specific LLM used as the reader.

Summary by alphaXiv.