Reproduce “Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Feng Luo, Yu-Neng Chuang, Guanchu Wang, Zicheng Xu, Xiaotian Han, Tianyi Zhang, Vladimir Braverman

Stable-OPD addresses instability in on-policy distillation, where student rollouts can become repetitive and truncated. It combines supervised training on complete example solutions with a reference-policy constraint. Across six math benchmarks, average accuracy rises from 28.9% to 36.1% in one comparison. Repetition and truncation coincide with performance changes, but the proposed feedback loop does not prove repetition alone causes collapse.

Key insights

  • The observed length inflation is driven by "abrupt repetition saturation," where student rollouts enter highly repetitive generation patterns.
  • Repetitive tokens systematically receive disproportionately larger reverse-KL advantage signals compared to regular tokens, creating a self-reinforcing feedback loop that encourages more repetition and length expansion.
  • This pathology causes severe training instability, characterized by sudden drops in validation accuracy and biased learning gradients due to truncation-dominated training.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.