Reproduce “ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Qingyu Zhang, Qianhao Yuan, Hongyu Lin, Yaojie Lu, Xianpei Han, Le Sun, Ming Xu, Jiarui Li

ShortOPD introduces a short-to-long on-policy distillation method for structured-pruned Large Language Models, dynamically managing the rollout budget to efficiently recover free-form generation capabilities. The approach restores up to 73.7% of the unpruned teacher's performance, achieving substantial gains over baselines while reducing training time and token consumption.

Key insights

  • Generative failures in pruned LLMs are primarily due to demoted correct trajectories and compounding local next-token mis-rankings, rather than erased knowledge, implying latent capabilities that can be recovered.
  • Effective recovery requires dense, token-level supervision applied to the specific on-policy states the compressed model visits during its own generation process.
  • Repetitive suffixes in early on-policy rollouts of heavily pruned models provide minimal distillation signal and consume significant computational resources, highlighting the need for dynamic budget management.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.