Reproduce “Thinking Isn't an Illusion: Overcoming the Limitations of Reasoning Models via Tool Augmentations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhao Song, Song Yue, Jiahao Zhang
Researchers from UC Berkeley and Northeastern University demonstrate that Large Reasoning Models (LRMs) consistently outperform non-reasoning LLMs on complex symbolic tasks when augmented with external tools like Python interpreters and scratchpads. The work challenges the "thinking-illusion" hypothesis by showing that tool-equipped LRMs can achieve near 100% accuracy on previously difficult puzzles like Hanoi Tower and substantial gains on River Crossing and Blocks World.
Key insights
- The previously observed "thinking illusion" in Large Reasoning Models (LRMs) was largely an artifact of evaluation conditions that constrained output length and tool access.
- External tool augmentation, particularly Python interpreters (Program-of-Thought) and scratchpads, is crucial for LRMs to externalize and manage reasoning steps, unlocking their full capabilities.
- The underlying strength of the base model influences how effectively it can leverage external tools, with stronger base models showing greater gains.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.