Reproduce “Not All LLM Reasoners Are Created Equal” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Arian Hosseini, Alessandro Sordoni, Daniel Toyama, Aaron Courville, Rishabh Agarwal

Researchers from Mila, Microsoft Research, and Google DeepMind introduced the 'Compositional GSM' benchmark and the 'reasoning gap' metric to evaluate LLMs' multi-step mathematical reasoning. They found that most LLMs exhibit a significant reasoning gap, performing worse on compositional problems than on individual ones, with code generation proving an effective strategy to mitigate this.

Key insights

  • Most LLMs, including specialized math models and smaller models, exhibit a 'reasoning gap,' performing worse on multi-step compositional problems than expected from their single-problem performance.
  • Instruction tuning improves overall performance but does not fully close the reasoning gap, especially for smaller models.
  • Finetuning on GSM8K can lead to overfitting, where models improve on the original benchmark but decline on compositional reasoning.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.