Reproduce “How Retrieved Context Shapes Internal Representations in RAG” on OpenResearch

About this paper

Samuel Yeh, Sharon Li

This paper explores how retrieved context, varying in relevance, influences the internal representations of large language models (LLMs) in Retrieval-Augmented Generation (RAG) systems. It reveals that random documents induce significant representation shifts correlating with model abstention, while relevant documents stabilize representations even amidst noise, and deep layers increasingly prioritize parametric knowledge, which limits RAG's impact on queries where external evidence is most crucial.

Key insights

• Random documents cause the most substantial shifts in LLM internal representations, directly correlating with increased abstention behavior, especially in instruction-tuned models, indicating an internal 'refusal' mechanism.
• LLMs exhibit robustness to noisy contexts; a single relevant document can stabilize internal representations and maintain accuracy even when surrounded by multiple distracting or random documents.
• Layer-wise processing shows that LLMs detect coarse semantic mismatches (random documents) in middle layers, while fine-grained distinctions and reconciliation with parametric knowledge occur in deeper layers, potentially limiting the impact of RAG on 'hard' queries.

Summary by alphaXiv.