Reproduce “On-Policy Parameter Update Direction Underlies Generalization in LLM Post-Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shufan Shen, Zhongni Hou, Junshu Sun, Yufei Zhang, Wei Lin, Guojun Yin, Qingming Huang, Shuhui Wang
The method constrains supervised fine-tuning using the signs of parameter changes learned during on-policy training. In the DeepMath ablation, full location-and-direction constraints reach a mean accuracy of 41.15, versus 34.07 with location constraints alone and 27.71 with random directions. This supports the role of direction, but does not show it is the only cause.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.