Reproduce “Prune&Comp: Free Lunch for Layer-Pruned LLMs via Iterative Pruning with Magnitude Compensation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xinrui Chen, Hongxing Zhang, Fanyi Zeng, Yongxian Wei, Yizhi Wang, Xitong Ling, Guanghao Li, Chun Yuan
Researchers at Tsinghua University developed PRUNE&COMP, a training-free and overhead-free method that addresses performance degradation in layer-pruned large language models by compensating for an identified "magnitude gap" in hidden states. This approach consistently improves perplexity and downstream task performance on benchmarks like MMLU and QA, enabling more effective model compression without additional computational costs.
Key insights
- Each individual Transformer layer inherently produces a significant "magnitude increase" in hidden states, leading to a pronounced "magnitude gap" when layers are removed.
- This intrinsic magnitude gap is a pervasive and overlooked cause of performance degradation in layer-pruned LLMs, regardless of the pruning metric used.
- Applying magnitude compensation during iterative pruning yields synergistic benefits by enhancing model stability and improving the precision of layer importance assessments.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.