Reproduce “Rethinking the Illusion of Thinking” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Iñaki Dellibarda Varela, Pablo Romero-Sorozabal, Eduardo Rocon, Manuel Cebrian
The paper "Rethinking the Illusion of Thinking" meticulously replicates and refines the controversial Towers of Hanoi and River Crossing benchmarks from Apple's "The Illusion of Thinking" study. It demonstrates that Large Reasoning Models (LRMs) excel at River Crossing when configurations are solvable but still struggle with the intrinsic complexity of Towers of Hanoi, even with stepwise prompting, offering a nuanced view of their reasoning capabilities.
Key insights
- LRM performance is highly dependent on task formulation, input validity, and prompt structure, rather than solely on intrinsic "reasoning" capacity.
- Failures previously attributed to LRM reasoning limitations, such as in River Crossing, can stem from evaluating mathematically unsolvable problem instances, highlighting the importance of rigorous benchmark design.
- LRMs often exhibit implicit uncertainty estimation, reducing token usage when a task is perceived as intractable, although this behavior can differ in collaborative agentic settings where persistent but unproductive effort is observed.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.