Reproduce “Beware of Calibration Data for Pruning Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yixin Ji, Yang Xiang, Juntao Li, Qingrong Xia, Ping Li, Xinyu Duan, Zhefeng Wang, Min Zhang
The paper systematically demonstrates that the choice of calibration data significantly impacts post-training large language model pruning performance, especially at high sparsity, and introduces a self-generating strategy that consistently outperforms existing calibration data sources, improving reasoning task performance by up to 2.68%.
Key insights
- Calibration data selection is a critical factor in post-training LLM pruning, with its impact becoming more pronounced at higher sparsity ratios and for structured pruning.
- Calibration data that is textually more similar to the LLM's original pre-training data consistently leads to better pruning performance than high-quality but dissimilar data.
- LLMs can effectively self-generate calibration data that reflects their internal training distribution, even outperforming directly sampled training data for pruning purposes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.