Reproduce “LightPAFF: A Two-Stage Distillation Framework for Pre-training and Fine-tuning” on OpenResearch

About this paper

Kaitao Song, Hao Sun, Xu Tan, Tao Qin, Jianfeng Lu, Hongzhi Liu, Tie-Yan Liu

Researchers at Nanjing University of Science and Technology, Peking University, and Microsoft Research Asia developed LightPAFF, a two-stage knowledge distillation framework for creating lightweight versions of large pre-trained language models. This framework reduces model parameter size by approximately 5 times and increases inference speed by 5 to 7 times while preserving nearly 99.5% of the teacher model's accuracy across various NLP tasks.

Key insights

• Distillation at both the pre-training and fine-tuning stages is essential to effectively transfer general linguistic knowledge and task-specific nuances, preventing significant accuracy drops.
• The optimal weight for the teacher's signal (`λ`) in distillation varies with the inherent difficulty of the pre-training task, with easier tasks allowing for higher reliance on teacher guidance.
• Student models trained with LightPAFF exhibit enhanced generalization abilities, demonstrating greater robustness to parameter perturbations, which implies convergence to more stable solutions.

Summary by alphaXiv.