Reproduce “LIMO: Less is More for Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, Pengfei Liu
The LIMO (Less is More for Reasoning) research from Shanghai Jiao Tong University and SII-GAIR develops a supervised fine-tuning method for large language models, demonstrating that advanced mathematical reasoning can be achieved with a minimal dataset of 800 carefully curated examples. This approach yields an average performance of 78.1% across various benchmarks, outperforming models like OpenAI-o1-preview (61.1%) and QwQ-32B-Preview (66.9%) that were fine-tuned on significantly larger datasets.
Key insights
- Complex reasoning in LLMs can emerge from minimal, high-quality data if foundation models possess comprehensive pre-trained domain knowledge.
- The quality of reasoning chains and the difficulty of training problems are critical factors in fostering robust and generalized reasoning abilities.
- A rich pre-trained knowledge base in the foundation model is essential for efficient elicitation of reasoning capabilities through fine-tuning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.