Reproduce “Learning Dynamics in RL Post-Training for Language Models” on OpenResearch

About this paper

Akiyoshi Tomihari

This research analyzes the learning dynamics of reinforcement learning (RL) post-training for language models using an empirical Neural Tangent Kernel (NTK) framework, revealing that RL systematically increases model confidence and reduces output diversity. It introduces Classifier-First Reinforcement Learning (CF-RL), a two-stage strategy that accelerates optimization by prioritizing classifier updates, leveraging the classifier's rapid adaptation in shaping effective RL dynamics.

Key insights

• RL post-training systematically increases model confidence and concentrates output probability distributions, leading to reduced output diversity, particularly driven by high-reward samples.
• The classifier parameters undergo the fastest relative change during RL training and play a critical role in shaping an informative Gradient component of the NTK.
• CF-RL accelerates optimization by reshaping the classifier's emphasis on structural tokens, a mechanism distinct from feature distortion reduction or classifier norm amplification seen in supervised linear probing.

Summary by alphaXiv.