Reproduce “Breaking the Compression Ceiling: Data-Free Pipeline for Ultra-Efficient Delta Compression” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiaohui Wang, Peng Ye, Chenyu Huang, Shenghe Zheng, Bo Zhang, Lei Bai, Wanli Ouyang, Tao Chen
UltraDelta is a data-free pipeline for compressing fine-tuned large language models and other neural networks, enabling ultra-high compression ratios while maintaining or improving performance. It achieved up to 224.6x compression for T5-base and 50.9x for LLaMA-2 13B, often outperforming uncompressed fine-tuned models across various benchmarks.
Key insights
- Layer variance correlates with information content, guiding a mixed sparsity allocation strategy to preserve critical information.
- Preserving intra-layer weight distributions through techniques like group-wise pruning is crucial for maintaining model performance.
- Adaptive rescaling factors, inversely proportional to delta weight trace norms, are essential for stabilizing models under extreme compression and can even offer regularization benefits for underfitted models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.