Reproduce “LoPRo: Enhancing Low-Rank Quantization via Permuted Block-Wise Rotation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hongyaoxing Gu, Lijuan Hu, Liye Yu, Haowei Li, Fangfang Liu
LOPRO introduces a fine-tuning-free Post-Training Quantization algorithm for Large Language Models, achieving optimal accuracy in sub-3-bit regimes by integrating importance-aware low-rank approximation with a permuted block-wise rotation for residual matrices. The method also features an efficient randomized SVD for faster decomposition, enabling high-performance LLM deployment with minimal overhead.
Key insights
- Carefully permuting and selectively rotating only a portion of the residual matrix (using an identity block for important columns) is crucial to reconcile the benefits of rotation with importance preservation in low-rank quantization.
- An iterative, simplified randomized SVD (R1SVD) can achieve comparable low-rank approximation accuracy to full SVD while offering significant speedup ($O(N^2)$ vs. $O(N^3)$ complexity) and reduced storage.
- Combining importance-aware scaling, low-rank approximation, and a specialized block-wise rotation strategy can effectively overcome the limitations of existing fine-tuning-free PTQ methods, especially in challenging ultra-low-bit scenarios.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.