Reproduce “Adaptive FastOPD: Progress-Aware Rollout Horizon Expansion for Efficient On-Policy Distillation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Qian Tan, Huaifei Liang, Xuanyu Zhu, Lei Jiang, Yuqiang Li
Researchers from the University of Science and Technology of China and collaborators developed Adaptive FastOPD, a method that dynamically adjusts rollout horizons for on-policy distillation (OPD) based on learning progress and resource utilization. This approach yields competitive or superior performance on mathematical reasoning benchmarks while reducing training time by 49.1% to 71.2% compared to full-horizon OPD baselines.
Key insights
- Dynamically adjusting the rollout horizon based on observed learning progress and resource utilization offers a more efficient and robust alternative to fixed-schedule or hyperparameter-sensitive OPD acceleration methods.
- Aggregating multiple student-teacher alignment signals with stage-relative normalization provides a comprehensive and adaptable measure for detecting when the student's learning has plateaued at a given sequence length.
- Incorporating a length-utilization gate is crucial for optimizing the trade-off between performance and efficiency, ensuring that horizon expansions contribute broadly to learning rather than just increasing costs for a few long sequences.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.