Reproduce “DRQA: Dynamic Reasoning Quota Allocation for Controlling Overthinking in Reasoning Large Language Models” on OpenResearch

About this paper

Kaiwen Yan, Xuanqing Shi, Hongcheng Guo, Wenxuan Wang, Zhuosheng Zhang, Chengwei Qin

Researchers developed DRQA, a reinforcement learning framework that enables reasoning large language models (RLLMs) to dynamically adjust their reasoning depth for single questions. The method mitigates "overthinking" by reducing token usage by an average of 30.4% while simultaneously improving reasoning accuracy by 1.58 percentage points across various benchmarks.

Key insights

• Batch inference naturally induces an 'resource competition pressure' in RLLMs, prompting them to generate significantly shorter and more concise reasoning chains with minimal accuracy degradation.
• Supervised fine-tuning to directly mimic concise outputs from batch inference leads to catastrophic forgetting, sacrificing deep reasoning capabilities for brevity.
• Training RLLMs with reinforcement learning to intrinsically differentiate and prefer reasoning chains that are both correct and concise allows for adaptive resource allocation without compromising performance.

Summary by alphaXiv.