Reproduce “Increasing the Thinking Budget is Not All You Need” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ignacio Iacobacci, Zhaozhi Qian, Faroq AL-Tam, Muhammad AL-Qurishi, Riad Souissi
Researchers at Elm Company investigated how Large Language Models (LLMs) utilize computational budgets across various reasoning strategies, demonstrating that simply increasing "thinking tokens" does not guarantee optimal performance. The study found that strategies like "Summary," which involves generating multiple diverse outputs and consolidating them, consistently outperform simpler configurations, achieving 80% accuracy on AIME24 with Qwen3-8B at a 6000-token budget, compared to 63.33% for the Vanilla approach.
Key insights
- Simply increasing the thinking budget does not guarantee optimal performance; the strategy for utilizing that budget is critical for achieving higher accuracy.
- More sophisticated strategies, particularly 'Summary,' consistently achieve more accurate responses and higher performance ceilings compared to simpler methods, even with a zero-thinking budget.
- Model capability influences the effectiveness of increased thinking budgets and complex strategies, with more capable models leveraging them to a greater extent.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.