Reproduce “On the Impact of Calibration Data in Post-training Quantization and Pruning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Miles Williams, Nikolaos Aletras
Researchers at the University of Sheffield conducted an extensive empirical study demonstrating that the choice of calibration data significantly impacts the downstream task performance of large language models after post-training quantization and pruning, contradicting the prevailing assumption of calibration data robustness. The study revealed performance variations of up to 9.4% in task accuracy between different calibration sets, with pruning methods generally exhibiting higher sensitivity than quantization.
Key insights
- The choice of calibration data significantly influences the downstream task performance of compressed LLMs, directly contradicting the prior assumption of broad robustness.
- Pruning methods, particularly SparseGPT, exhibit greater sensitivity and performance dispersion to calibration data compared to quantization methods.
- There is no universally "optimal" calibration data source, as the impact varies depending on the specific compression method and LLM architecture.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.