Reproduce “On the Role of Reasoning Patterns in the Generalization Discrepancy of Long Chain-of-Thought Supervised Fine-Tuning” on OpenResearch

About this paper

Zhaoyi Li, Xiangyu Xi, Zhengyu Chen, Wei Wang, Gangwei Jiang, Ranran Shen, Linqi Song, Ying Wei and 1 more

An investigation into Long Chain-of-Thought (CoT) Supervised Fine-Tuning (SFT) reveals that structural reasoning patterns in teacher-generated trajectories influence student model generalization, even when training loss is lower. Student models fine-tuned on more exploratory, redundant CoT data from DeepSeek-R1 exhibited inferior generalization compared to those trained on more deductive CoT from gpt-oss-120b. Filtering for less branching reasoning patterns improved student model performance by up to 2.2% average accuracy.

Key insights

• DeepSeek-R1's CoT trajectories, while achieving lower SFT training loss, contained a 'dilution effect' due to a higher proportion of routine tokens and an excess of exploratory ('Propose') reasoning steps, leading to structural redundancy.
• Student models fine-tuned on DeepSeek-R1 data inherited these highly exploratory reasoning patterns, resulting in frequent 'Propose Propose' transitions that often trapped them in redundant branches during inference, thereby hindering convergence and generalization.
• The observed performance gap was not due to training duration, output truncation, or factual incorrectness, but rather to the intrinsic structural patterns of reasoning, with deductive reasoning (gpt-oss) proving more effective for generalization than extensive exploration (DeepSeek-R1).

Summary by alphaXiv.