Reproduce “ReEfBench: Quantifying the Reasoning Efficiency of LLMs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhizhang Fu, Yuancheng Gu, Chenkai Hu, Hanmeng Liu, Yue Zhang

ReEfBench, a neuro-symbolic framework developed by researchers from Westlake University, Imperial College London, NYU, and Hainan University, quantifies the reasoning efficiency of Large Language Models by formally grounding natural language responses in First-Order Logic. The system identifies four distinct reasoning behavioral archetypes and reveals that modern short Chain-of-Thought models can achieve logical depths comparable to longer variants with reduced computational cost.

Key insights

  • Four distinct LLM reasoning archetypes (Deep Wanderer, Effective Solver, Hollow Mimic, Lazy Guesser) are identified, providing a diagnostic taxonomy for understanding how models succeed or fail.
  • Modern Short Chain-of-Thought (CoT) models, especially with reflection mechanisms, can achieve logical depths comparable to Long CoT with significantly reduced computational cost, challenging the assumption that deep reasoning requires extensive token generation.
  • Mixing Long and Short CoT training data can lead to strategy interference and premature saturation in LLMs, while distillation into smaller models often results in 'behavioral mimicry' without genuine logical efficacy due to parametric capacity limits.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.