Reproduce “Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shiwan Zhao, Zhihu Wang, Xuyang Zhao, Jiaming Zhou, Caiyue Xu, Chenfei Liu, Liting Zhang, Yuhang Jia and 5 more
A unified framework is proposed for Large Language Model post-training, categorizing methods by trajectory provenance (off-policy or on-policy) and functional roles like effective support expansion, policy reshaping, and behavioral consolidation. This framework offers a systematic lens to understand, compare, and design complex multi-stage pipelines, highlighting key bottlenecks and future research challenges in developing aligned LLMs.
Key insights
- Trajectory provenance (off-policy vs. on-policy) is the most informative distinction, dictating which states the learner is trained on and what failures are directly observed.
- Hybrid post-training pipelines are best understood as coordinated multi-stage systems, where each stage addresses specific behavioral bottlenecks (expansion, reshaping, consolidation).
- Distillation primarily serves "behavioral consolidation," transferring and stabilizing useful behaviors from complex or fragile sources into more deployable models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.