Reproduce “Rethinking Easy-to-Hard: Limits of Curriculum Learning in Post-Training for Deductive Reasoning” on OpenResearch

About this paper

Maximilian Mordig, Andreas Opedal, Weiyang Liu, Bernhard Schölkopf

An empirical study found that difficulty-based curriculum learning strategies provided no robust advantage over random sampling for post-training large language models (LLMs) on deductive reasoning tasks, challenging common pedagogical intuition. The research observed that reinforcement learning (RL) generally improved out-of-distribution accuracy more effectively than supervised fine-tuning (SFT).

Key insights

• Difficulty-based curriculum learning, regardless of sequencing strategy, did not consistently or robustly enhance LLM accuracy or response length compared to random sampling during post-training.
• Reinforcement learning (RL) generally fostered better out-of-distribution generalization and accuracy than supervised fine-tuning (SFT), suggesting that robust feedback mechanisms might be more crucial than data sequencing.
• The empirical distributions of response lengths were similar across all curriculum strategies, implying that example ordering did not significantly alter the models' effective reasoning depth.

Summary by alphaXiv.