Reproduce “Less is More: Early Stopping Rollout for On-Policy Distillation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhou Ziheng, Jiaqi Li, Huacong Tang, Ying Nian Wu, Demetri Terzopoulos

A method called Early Stopping Rollout (ESR) refines on-policy distillation for large language models by concentrating supervision on initial tokens, which addresses the issue of "Off-policy Teacher Decay." This leads to substantial improvements in training efficiency, greater stability across various model pairs, and instances where student models can even outperform their teachers.

Key insights

  • "Off-policy Teacher Decay" empirically demonstrates that teacher feedback becomes less reliable as student-generated prefixes grow longer.
  • Training on early, strategic tokens induces a "Cascading Alignment Effect," where students infer and apply the teacher's reasoning strategies to later, untrained parts of the sequence.
  • Early Stopping Rollout facilitates "Sub-mode Commitment," allowing students to discover and adopt more concise or efficient problem-solving strategies, sometimes outperforming the teacher.
  • Token position serves as a load-bearing, independent axis for effective distillation supervision, distinct from scalar saliency metrics like KL divergence or entropy.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.