Reproduce “Examining False Positives under Inference Scaling for Mathematical Reasoning” on OpenResearch

About this paper

Yu Wang, Nan Yang, Liang Wang, Furu Wei, Fuli Feng

This research systematically investigates "false positives" in large language models' mathematical reasoning, where correct final answers are derived from flawed intermediate steps, particularly under inference scaling techniques. It finds that these reasoning errors persist even with increased computational resources, impacting standard evaluation metrics and suggesting a lower true ceiling for reasoning performance.

Key insights

• Current LLMs, including powerful models like GPT-4o, demonstrate limited effectiveness in reliably detecting errors in their own intermediate reasoning steps, underscoring the necessity of human evaluation for robust assessment.
• False positives persist across all tested inference scaling methods, indicating that merely increasing computational resources during inference does not inherently mitigate underlying reasoning flaws.
• The Pass@N metric, commonly used in automatic evaluations, is highly susceptible to false positives, suggesting that the actual ceiling for true reasoning performance under inference scaling is lower than automatic metrics might indicate.

Summary by alphaXiv.