Reproduce “Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap” on OpenResearch

About this paper

Saurabh Srivastava, Annarose M B, Anto P V, Shashank Menon, Ajay Sukumar, Adwaith Samod T, Alan Philipose, Stevin Prince and 1 more

Researchers from Consequent AI introduced a framework to quantify the 'reasoning gap' in Large Language Models, which is the performance difference between static benchmarks and functionally equivalent problem variants. Their evaluation showed that even advanced LLMs exhibit substantial reasoning gaps, ranging from 58.35% to 80.31% on the MATH dataset, indicating an overestimation of true reasoning capabilities by current evaluation methods.

Key insights

• All evaluated LLMs, including state-of-the-art models like GPT-4, exhibit substantial reasoning gaps, indicating a reliance on memorization rather than genuine reasoning.
• The reasoning gap increases with problem difficulty, suggesting models struggle more with genuine reasoning on harder problems.
• The functional benchmark approach provides a more accurate assessment of LLM reasoning abilities, guiding future research toward models with stronger generalization.

Summary by alphaXiv.