Reproduce “How and Why LLMs Generalize: A Fine-Grained Analysis of LLM Reasoning from Cognitive Behaviors to Low-Level Patterns” on OpenResearch

About this paper

Haoyue Bai, Yiyou Sun, Wenjie Hu, Shi Qiu, Maggie Ziyu Huan, Peiyang Song, Robert Nowak, Dawn Song

Researchers from the University of Wisconsin, Madison, UC Berkeley, University of Pennsylvania, and Caltech investigated how Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) impact LLM generalization by decomposing reasoning into five atomic cognitive skills. Their findings indicate RL maintains balanced skill profiles and better cross-domain transfer, while SFT often leads to over-specialization and skill degradation, irrespective of similar parameter modification magnitudes.

Key insights

• LLM reasoning emerges from a composition of atomic core skills, which are distinctly shaped and distributed by different post-training strategies.
• The specific optimization objective of a post-training strategy (SFT vs. RL) is more influential in determining an LLM's cognitive profile and generalization than the sheer scale of parameter modifications.
• Richer, explicit Chain-of-Thought (CoT) supervision during SFT can mitigate over-specialization and foster more balanced skill profiles by encouraging the internalization of reasoning processes.

Summary by alphaXiv.