Reproduce “NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densities” on OpenResearch

About this paper

Mo Li, Songyang Zhang, Taolin Zhang, Haodong Duan, Yunxin Liu, Kai Chen

The NeedleBench framework, developed by researchers from Tsinghua University and Shanghai AI Laboratory, provides a bilingual synthetic benchmark to rigorously evaluate large language models' retrieval and multi-step reasoning abilities across varying information densities. It reveals that although models demonstrate strong retrieval capabilities, their performance on information-dense, multi-step reasoning tasks degrades significantly, often failing beyond 64-needle levels, with "under-thinking" identified as a prevalent error.

Key insights

• Synthetic task design is crucial for accurate evaluation, as models' performance on synthetic reasoning tasks is 50-80% lower than on realistic, memorization-prone tasks.
• While LLMs demonstrate strong retrieval capabilities in information-sparse contexts, their ability to perform multi-step, information-dense reasoning in long contexts remains severely limited.
• "Under-thinking," where models prematurely cease inference despite relevant contextual information, is identified as a primary failure mode, accounting for 60.2% of errors in strong models.

Summary by alphaXiv.