Reproduce “Perturbing Inputs Improves Pretraining Distillation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ziyang Cai, Yihe Dong, Dimitris Papailiopoulos, Shivam Garg

The authors test whether distillation improves when a student matches a teacher’s predictions on perturbed inputs before returning to clean-text training. They examine token noise and blending, including neighboring-token blends. For a 0.5B Qwen2.5 student, equal-weight adjacent blending lowers clean held-out validation loss by 0.020 nats across three seeds; evidence covers only tested sizes and substantially longer training.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.