Reproduce “SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V. Le, Sergey Levine and 1 more
The authors compare reinforcement learning (RL) with supervised fine-tuning (SFT) on rule and visual changes in GeneralPoints and virtual navigation. RL uses verifier feedback through attempts, while SFT learns from expert responses. On held-out variants, RL generally transfers better; on a visual navigation benchmark, RL reaches 77.8% route success versus 11.1% for SFT, in these tested settings.
Key insights
- Reinforcement Learning fosters generalizable knowledge, enabling models to adapt to novel rules and unseen visual contexts.
- Supervised Fine-Tuning primarily leads to memorization of training data and struggles with out-of-distribution generalization.
- SFT serves as a necessary prerequisite, acting as a "format teacher" to stabilize model outputs for effective subsequent RL training, while more verification steps in RL significantly enhance generalization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.