Reproduce “Pruning Foundation Models for High Accuracy without Retraining” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Pu Zhao, Fei Sun, Xuan Shen, Pinrui Yu, Zhenglun Kong, Yanzhi Wang, Xue Lin
Researchers from Northeastern University and Meta introduce a post-training pruning approach for foundation models that formulates pruning as a "Multiple Removal Problem" (MRP), achieving lower perplexity and higher zero-shot accuracy at high sparsity levels without costly retraining. This method consistently outperforms prior post-training pruning techniques by explicitly accounting for interactions between simultaneously removed weights.
Key insights
- Treating pruning as a Multiple Removal Problem (MRP) that considers simultaneous weight interactions yields better results than sequential, independent weight removal (SRP).
- It is possible to derive an optimal compensation for remaining weights after multiple removals without full retraining.
- The proposed framework is versatile, applicable to various LLM architectures (Transformer, Mamba) and both unstructured and semi-structured pruning configurations.
- Lower dampening ratios and sufficient calibration samples (around 400) contribute to better pruning performance in the MRP approach.
- The MM algorithm variant, applying MRP for both mask selection and compensation, generally achieves the best performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.