Reproduce “Understanding Knowledge Distillation in Post-Training: When It Helps and When It Fails” on OpenResearch

About this paper

Xin Liu, Simin Ma, Shujian Liu, Song Wang, Sathish Reddy Indurthi, Haoyun Deng, Lu Wang, Kaiqiang Song

The research systematically investigates Knowledge Distillation (KD) in the post-training stage of Large Language Models (LLMs), revealing its effectiveness in low-data environments and with stronger teachers. It proposes a two-stage synthetic data strategy that consistently improves student model performance in low-resource, domain-specific tasks, achieving up to 5% absolute performance gains over supervised fine-tuning in limited data settings.

Key insights

• KD's effectiveness is strongly dependent on data availability, providing its greatest utility in low-data regimes where it can substantially outperform direct supervised fine-tuning.
• Distilling from a stronger, more broadly instruction-tuned teacher model yields considerable performance gains, even with abundant training data, by transferring novel knowledge or reasoning strategies not easily acquired through direct supervision alone.
• For domain-specific, low-resource scenarios, a two-stage KD approach that first warms up the student on synthetic data and then refines it with GKD on human annotations proves most effective.

Summary by alphaXiv.