Reproduce “Sparsity Induction for Accurate Post-Training Pruning of Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Minhao Jiang, Zhikai Li, Xuewen Liu, Jing Zhang, Mengjuan Chen, Qingyi Gu
The Sparsity Induction (SI) framework, developed by the Institute of Automation, Chinese Academy of Sciences, proactively prepares large language models for pruning by shaping their weight distribution and feature structure. This method consistently improves accuracy for various post-training pruning techniques at aggressive sparsity levels while introducing no inference-time overhead.
Key insights
- Existing post-training pruning methods primarily refine importance scores, yielding marginal gains because they don't address the fundamental disruption to model states.
- Pruning dense LLMs damages the self-consistent distributional and scaling structures established during pre-training, leading to significant accuracy degradation.
- Proactively inducing sparsity-friendly distributions and feature structures *before* pruning is more effective than reactive compensation, allowing for higher sparsity without severe accuracy loss.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.