Reproduce “Layer-wise Curriculum Learning for Efficient LLM Compression” on OpenResearch

About this paper

Donggeon Lee, Dooyeon Na, Seungmin Oh, Jongbin Ryu

Layer-wise Curriculum Learning for Efficient LLM Compression combines local knowledge distillation with a depth-based training schedule that progressively freezes shallow layers and concentrates optimization on deeper ones, where representation errors accumulate. Across BERT, GPT-2, LLaMA, and Qwen models, the method lowers training and memory costs while generally preserving or improving compressed-model quality, including 80.0 average GLUE for BERT and higher pruning benchmarks than competing methods at comparable compression levels.

Key insights

• Compression quality depends not only on which layers are removed but also on how optimization is allocated across depth; deeper layers require more attention because they inherit upstream representation errors.
• Updating student segments with their own propagated features, rather than relying exclusively on teacher inputs, better reflects the conditions encountered during deployment.
• The curriculum and caching components are complementary: on LLaMA3-8B, their combination improved the reported average score from 56.9 for the baseline to 59.5, while cosine loss outperformed MSE, normalized MSE, and KL-divergence alternatives.

Summary by alphaXiv.