Reproduce “When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuxiao Yang, Tianrun Yu, Shangzhe Li, Kaixiang Zhao, Xuchao Zhang, Chetan Bansal, Huaxiu Yao, Taylor W. Killian and 1 more
The authors study length inflation in on-policy distillation (OPD), where a student learns from its own sampled responses and a teacher’s token-level guidance. They show that student and teacher may represent stopping with different equivalent tokens, causing the signal to suppress termination. Aggregating stopping probability mitigates inflation across Qwen3, Llama 3.2, and Gemma 3, but not late K2-Horizon re-inflation.
Key insights
- Recognizing multiple EOS tokens at decoding time does not resolve the problem because the distillation loss still treats equivalent termination tokens as separate actions.
- In Qwen3, the student’s native EOS probability falls from roughly 0.8 to near zero, while the teacher-preferred token has student probability around 10^-11, creating a strongly asymmetric learning signal.
- Semantic termination alignment transfers stopping behavior across model families, but K2-Horizon experiments show that late-stage length growth can remain even when EOS mismatch is removed.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.