Reproduce “ThinkDial: An Open Recipe for Controlling Reasoning Effort in Large Language Models” on OpenResearch

About this paper

Qianyu He, Siyu Yuan, Xuefeng Li, Mingxuan Wang, Jiangjie Chen

ThinkDial introduces the first open-recipe, end-to-end framework for large language models to control reasoning effort through discrete operational modes, akin to proprietary systems. This framework consistently achieves targeted reasoning compression and performance trade-offs across diverse tasks, enabling dynamic allocation of computational resources based on user preference.

Key insights

• Budget-Mode Supervised Fine-tuning (SFT) is essential for establishing stable, mode-specific reasoning patterns and preventing performance collapse during subsequent RL.
• A two-phase Reinforcement Learning approach, incorporating a "warm-up" and budget-aware reward shaping with a "Leak Penalty," is critical for effective and genuine reasoning compression without performance loss.
• Over-aggressive SFT data or naive truncation methods severely degrade performance, highlighting the need for a sophisticated, learned compression mechanism.

Summary by alphaXiv.