Reproduce “ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidence” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kevin Wu, Eric Wu, James Zou
ClashEval tests whether language models arbitrate between prior answers and retrieved context when either may be wrong. The benchmark covers over 1,200 questions, perturbing context answers and measuring model choices. GPT-4o chose erroneous context answers in 60.8% of cases where its prior was correct. A confidence-based correction improved conditional accuracy, but results do not establish performance in deployed RAG systems.
Key insights
- LLMs, including GPT-4o, demonstrate a high susceptibility to incorrect retrieved content, overriding their own correct prior knowledge over 60% of the time.
- A clear inverse relationship exists between the degree of contextual modification (deviation from truth) and an LLM's likelihood of adopting that context, indicating a basic plausibility check.
- For models providing token probabilities, a consistent inverse correlation was found between the confidence in an LLM's prior answer and its tendency to defer to external context, suggesting a rudimentary internal calibration mechanism.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.