Reproduce “Investigating Context-Faithfulness in Large Language Models: The Roles of Memory Strength and Evidence Style” on OpenResearch

About this paper

Yuepei Li, Kang Zhou, Qiao Qiao, Bach Nguyen, Qing Wang, Qi Li

Researchers at Iowa State University investigated how Large Language Models manage conflicting information by quantifying LLM memory strength and exploring different evidence presentation styles. They found that stronger internal memory reduces context faithfulness, while presenting external evidence through paraphrasing significantly improves LLMs' receptiveness to new facts.

Key insights

• An LLM's internal memory strength is a primary determinant of its context faithfulness, positively correlating with its tendency to rely on internal knowledge.
• Evaluation metrics for context faithfulness need to be memory strength-aware to provide fair comparisons and avoid penalizing more knowledgeable models.
• Linguistic variability, specifically presenting multiple paraphrased versions of evidence, is significantly more effective than simple repetition or adding details for improving LLM receptiveness to conflicting external information.

Summary by alphaXiv.