Reproduce “Less is More Tokens: Efficient Math Reasoning via Difficulty-Aware Chain-of-Thought Distillation” on OpenResearch

About this paper

Abdul Waheed, Chancharik Mitra, Laurie Z. Wang, Deva Ramanan, Bhiksha Raj

Carnegie Mellon University researchers developed a framework for Large Language Models to dynamically adjust their Chain-of-Thought reasoning length based on problem difficulty. This approach reduces reasoning token usage by up to 30-35% while maintaining or improving accuracy across unimodal and multimodal math problems.

Key insights

• LLMs can be trained to internally adapt reasoning depth based on problem difficulty without requiring architectural modifications, mimicking human cognitive processes.
• Supervised Fine-Tuning (SFT) effectively teaches models to generate concise and structured reasoning patterns, while Direct Preference Optimization (DPO) is crucial for preserving and improving reasoning accuracy.
• A hybrid SFT+DPO training strategy synergistically combines these benefits, achieving substantial token efficiency while maintaining or improving final answer accuracy.

Summary by alphaXiv.