Reproduce “Prompt Curriculum Learning for Efficient LLM Post-Training” on OpenResearch

About this paper

Zhaolin Gao, Joongwon Kim, Wen Sun, Thorsten Joachims, Sid Wang, Richard Yuanzhe Pang, Liang Tan

Researchers from Meta Superintelligence Labs and Cornell University developed Prompt Curriculum Learning (PCL), an efficient framework for post-training large language models with reinforcement learning. PCL uses a learned value model to dynamically select prompts of intermediate difficulty, which accelerates training by up to 16.9x in filtering time while achieving higher performance.

Key insights

• An optimal batch size exists where generation time scales sublinearly, leading to maximally fast convergence for LLM RL.
• Prompts where the current policy achieves approximately 50% success yield the highest effective learning signals and gradient magnitudes.
• A learned value model can accurately predict prompt difficulty comparable to multiple generations, enabling efficient on-policy curriculum learning without wasteful rollouts.

Summary by alphaXiv.