Reproduce “On Predictability of Reinforcement Learning Dynamics for Large Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuchen Cai, Ding Cao, Xin Xu, Zijun Yao, Yuqing Huang, Zhenyu Tan, Benyi Zhang, Guangzhong Sun and 2 more

This research analyzes how large language models evolve during reinforcement learning (RL) training, revealing that parameter updates are dominated by a single, linearly evolving low-rank subspace. Leveraging these findings, the AlphaRL framework accelerates RL training by up to 2.5 times, preserving over 96% of the model's final reasoning performance.

Key insights

  • RL-induced parameter updates exhibit 'Rank-1 Dominance,' where the single largest singular subspace of the update matrix (ΔW) recovers over 99% of reasoning performance gains.
  • The dominant Rank-1 subspace evolves with 'Rank-1 Linear Dynamics,' demonstrating a highly predictable linear trajectory throughout RL training, enabling accurate performance extrapolation.
  • Unlike SFT and DIST, RL updates are characterized by smaller magnitude, higher concentration in low-rank subspaces, and minimal shifts in embedding space, indicating targeted high-level reasoning optimization.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.