Reproduce “Reinforcement Mid-Training” on OpenResearch

About this paper

Yijun Tian, Shaoyu Chen, Zhichao Xu, Yawei Wang, Jinhe Bi, Peng Han, Wei Wang

Reinforcement Mid-Training (RMT) formalizes a critical third stage in large language model development, applying reinforcement learning on unlabeled pre-training data to systematically enhance complex reasoning capabilities. The method achieves up to +64.91% higher language modeling accuracy compared to prior RL-based mid-training approaches while reducing reasoning response length by up to 79%.

Key insights

• Integrating reinforcement learning into a dedicated 'mid-training' stage between pre-training and post-training provides a principled way to build complex reasoning skills from unlabeled data.
• Dynamic control over reasoning length via a token budget and specific reward shaping significantly improves computational efficiency without compromising model effectiveness.
• An adaptive curriculum that prioritizes tokens based on their learning difficulty, coupled with a dual training objective, enables stable learning and comprehensive utilization of all token information.

Summary by alphaXiv.