Reproduce “ROSAQ: Rotation-based Saliency-Aware Weight Quantization for Efficiently Compressing Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Junho Yoon, Geom Lee, Donghyeon Jeon, Inho Kang, Seung-Hoon Na
ROSAQ, a post-training quantization method, leverages rotational invariance and Principal Component Analysis to identify and preserve salient channels in Large Language Models. It applies a mixed-precision strategy, retaining FP16 for critical channels and quantizing others to INT3/INT4, achieving competitive performance and up to 2.3x inference speedup on LLaMA and Qwen models.
Key insights
- Rotational invariance can be effectively leveraged through PCA to reveal statistically more dominant and informative channels for saliency identification, distinct from magnitude-based approaches.
- Applying PCA in a 'head-wise' manner for Multi-Head Self-Attention layers better captures the unique characteristics of individual attention heads, improving quantization outcomes.
- Preserving higher precision only for PCA-identified salient channels, combined with aggressive low-precision quantization for non-salient ones, effectively balances compression with performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.