Reproduce “CoverBench: A Challenging Benchmark for Complex Claim Verification” on OpenResearch

About this paper

Alon Jacovi, Moran Ambar, Eyal Ben-David, Uri Shaham, Amir Feder, Mor Geva, Dror Marcus, Avi Caciularu

Google Research introduced CoverBench, a challenging benchmark designed for complex claim verification, revealing that even state-of-the-art large language models struggle significantly, often performing near random chance on tasks requiring multi-faceted reasoning across diverse domains. The benchmark's rigorous design effectively exposes the current limitations of LLMs in accurately verifying information in complex, real-world scenarios.

Key insights

• CoverBench provides a robust and challenging evaluation, demonstrating significant weaknesses in current LLMs' ability to accurately verify information requiring deep, multi-faceted reasoning across diverse data types and domains.
• The benchmark's selection methodology effectively isolates truly difficult examples, validating its design for identifying genuine complex reasoning challenges rather than relying on superficial cues.
• Chain-of-Thought prompting offers only marginal performance gains, and few-shot learning struggles in long-context scenarios, indicating that current methods for explicit reasoning do not fundamentally bridge the gap in core verification capabilities.

Summary by alphaXiv.