Reproduce “Trust, but Don't Verify: Epistemic Blind Spots in LLM Source Evaluation” on OpenResearch

About this paper

Rohan N. Pradhan, Steve Goley

This research reveals that large language models (LLMs) exhibit an 'epistemic blind spot,' effectively crediting statistically impossible numerical claims when presented within a plausible methodological context during multi-source synthesis. While LLMs can detect these flaws in isolation, they generally fail to deploy this critical discernment in conversational settings, retaining about 74% of the influence of valid statistics even for fabricated numbers.

Key insights

• LLMs possess the latent ability to detect statistical impossibilities when claims are presented in isolation, but this capability largely collapses when claims are embedded in multi-source synthesis tasks.
• The 'methodology-register gate' strongly influences LLM trust: a plausible analytical description significantly boosts a source's influence, regardless of the underlying statistical validity of its claims.
• Current post-training pipelines (SFT, DPO, RLVR) enhance sensitivity to the aesthetics of rigor but do not develop robust internal mechanisms for verifying the consistency and validity of numerical data.

Summary by alphaXiv.