Reproduce “Incentivizing Strong Reasoning from Weak Supervision” on OpenResearch

About this paper

Yige Yuan, Teng Xiao, Shuchang Tao, Xue Wang, Jinyang Gao, Bolin Ding, Bingbing Xu

Researchers introduce a "Weak-to-Strong Reasoning (W2SR)" paradigm, where stronger Large Language Models are fine-tuned using Chain-of-Thought trajectories generated by demonstrably weaker models. This method recovers up to 94.34% of reasoning gains typically achieved by expensive reinforcement learning (RL), while also demonstrating a 10x to 25x reduction in training time and inducing enhanced cognitive behaviors.

Key insights

• The teacher model's explicit reasoning ability (i.e., its capacity to produce structured CoT traces) is more critical for effective student supervision than its overall model scale or final answer accuracy.
• Reasoning traces, even those leading to an incorrect final answer (W2SR-N), provide valuable learning signals, with students trained on incorrect CoT outperforming vanilla students.
• The structural soundness and intermediate steps of the reasoning process are pedagogically beneficial, enabling students to learn structured thinking and correct computational errors independently.

Summary by alphaXiv.