Reproduce “Can LLMs Reconcile Knowledge Conflicts in Counterfactual Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Khurram Yamin, Gaurav Ghosal, Bryan Wilder
A study from Carnegie Mellon University systematically probes Large Language Models' capacity to reconcile internal knowledge with in-context counterfactual premises for multi-hop reasoning. It reveals that LLMs frequently exhibit context-ignoring or context-overfitting behaviors, achieving near-random performance on conflicting information scenarios, with post-hoc fine-tuning offering marginal benefits.
Key insights
- Production LLMs demonstrate a strong bias towards parametric knowledge, leading to difficulties (near 50% accuracy) in overriding conflicting contextual information in multi-hop counterfactual scenarios.
- Post-hoc fine-tuning for counterfactual reasoning often leads to models learning superficial heuristics rather than robust selective override mechanisms, and can exacerbate the degradation of previously stored parametric knowledge.
- The challenge lies in the dynamic integration of conflicting knowledge, not in general catastrophic forgetting or simple format adaptation, as shown by rapid learning on factual Chain-of-Thought tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.