Reproduce “An RL View of OPD: Least Square Policy Distillation for Sample-Efficient LLM Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shangzhe Li, Yuxiao Yang, Tianrun Yu, Kaixiang Zhao, Xiaoyun Wang, Taylor W. Killian, Weitong Zhang
LSPD trains smaller language models with token-level feedback from a stronger teacher on student-generated solutions, reusing earlier trajectories with a replay buffer. It combines robust log-probability matching with an entropy bonus. Across 18 model–benchmark combinations, LSPD exceeds EOPD by 0.91 Avg@16 points and 0.85 Pass@16 points. LSPD-RB reaches its plateau in about 10 rollout batches, not less time or compute.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.