Reproduce “Source Attribution in Retrieval-Augmented Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ikhtiyor Nematov, Tarik Kalai, Elizaveta Kuzmenko, Gabriele Fugagnoli, Dimitris Sacharidis, Katja Hose, Tomer Sagi

Researchers investigated source attribution in Retrieval-Augmented Generation (RAG) by adapting game-theoretic Shapley values to identify influential documents from a retrieved set. They defined document utility as the Large Language Model's (LLM) log-likelihood for generating a target response, evaluating various approximation methods and revealing their effectiveness and inherent biases in complex document interactions.

Key insights

  • Kernel SHAP and ContextCite consistently provide accurate and efficient approximations of Shapley values for document-level attribution in RAG, making practical explainability feasible under computational constraints.
  • Utility-based attribution methods exhibit a significant position bias, consistently assigning higher influence scores to documents appearing earlier in the context, even when information is redundant.
  • These methods struggle with multi-hop reasoning (synergy) scenarios, systematically underestimating the contribution of prerequisite documents that enable the answer while overcrediting documents containing direct answer segments.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.