Reproduce “On the Implicit Reward Overfitting and the Low-rank Dynamics in RLVR” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hao Ye, Jisheng Dang, Junfeng Fang, Bimei Wang, Yizhou Zhang, Ning Lv, Wencan Zhang, Hong Peng and 2 more
This research reveals that Reinforcement Learning with Verifiable Rewards (RLVR) in Large Language Models implicitly overfits training rewards, with non-dominant weight update components inflating training performance without significantly improving generalization. The study identifies a universal "leading spike, heavy-tailed" spectral pattern in RLVR-induced weight updates, showing that while rank-1 components drive reasoning, higher-rank components are crucial for preserving general knowledge and safety, and geometrically, LoRA adapters align asymmetrically, prioritizing output-side optimization.
Key insights
- RLVR exhibits "implicit reward overfitting" where non-Rank-1 components primarily inflate training rewards without translating into proportional improvements in generalized test-set accuracy.
- Non-Rank-1 components are functionally crucial for preserving the model's general knowledge, safety, and instruction-following abilities, mitigating catastrophic forgetting during RLVR.
- RLVR-induced weight updates consistently display a universal "leading spike, heavy-tailed" singular spectrum pattern across linear layers, indicating structured, non-stochastic contributions from even low-amplitude components.
- There is a geometric asymmetry in rank-1 LoRA training dynamics, with the output-side component aligning more strongly with the principal left singular subspace, suggesting RLVR primarily optimizes output/sampling efficiency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.