Reproduce “When Text and Numbers Disagree: Evidence Arbitration in Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mattia Carletti, Edward Phillips, Fredrik K. Gustafsson, Patitapaban Palo, Lei Clifton, Danielle Belgrave, Xiao Gu, David A. Clifton
The authors introduce a synthetic benchmark testing how language models arbitrate between conflicting text and numerical evidence in binary risk predictions. They create controlled conflicts varying modality, recency, reliability, order, and simulated forecasts. Models show systematic source preferences, and some are strongly misled by false forecasts, especially when correct context appears first. The benchmark does not establish real-world performance.
Key insights
- LLMs do not resolve evidence conflicts randomly; they employ systematic and often heuristic arbitration strategies influenced by salient evidence characteristics.
- Temporal recency serves as a more consistent and influential arbitration cue than explicit source reliability, indicating a preference for newer information.
- A strong prompt recency effect exists, where the position of evidence in the prompt significantly influences LLM arbitration decisions.
- Models exhibit a pronounced over-reliance on external tool forecasts, even when these forecasts are systematically contradicted by accurate contextual evidence.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.