Reproduce “Evidence Integration in Large Language Models” on OpenResearch

About this paper

Sebastien Kawada, Manolis Kellis

The authors study how models integrate supplied answers, tool results, and other evidence after forming an initial answer distribution. Their evidence-integration law separates prior weight from evidence tilt and distinguishes checking from using. In synthetic tasks across twelve models, Qwen3-32B rejected every violating SAT assignment yet adopted 100% of them in 13,716 conditional trials; deployment prevalence remains unknown.

Key insights

• Internal support is a major determinant of adoption: models preferentially accept candidates resembling their own characteristic errors, and fitted prior weights fall below the rational Bayesian benchmark across all nine arithmetic models.
• Evidence quality is receiver-relative. The same source or near-miss candidate can help on problems where a model is weak but harm it on problems it would otherwise solve, so source accuracy alone is insufficient for evaluating tool reliability.
• Verification is not equivalent to control: models often reject a candidate in isolation yet adopt it when using it to answer, while mechanistic analyses place candidate integration late in processing and show that verbalized verdicts may not causally govern decisions.

Summary by alphaXiv.