Reproduce “IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compression” on OpenResearch

About this paper

Ali Abbasi, Chayne Thrash, Haoran Qin, Hamed Pirsiavash, Soheil Kolouri

IO-SVD introduces a post-training compression method for large language and vision-language models that utilizes KL-aware input-output whitened SVD, adaptive rank allocation, and loss-aware remapping for hybrid SVD-quantization. The approach improves model quality under aggressive compression, yielding up to 4.34x faster decoding and a 3.3x memory reduction for a LLaMA-2-7B model while maintaining high accuracy across various benchmarks.

Key insights

• Incorporating both input activation geometry and output-side predictive sensitivity via a KL divergence-derived double-sided whitening space leads to more robust low-rank approximations than input-only methods.
• Adaptive, heterogeneous rank allocation, guided by the sensitivity of individual singular components to calibration loss, more effectively distributes compression capacity across different layers than fixed or uniform strategies.
• Selective, loss-aware quantization of low-rank factors can further mitigate performance losses incurred during aggressive SVD truncation, enabling robust hybrid compression even at very high compression ratios.

Summary by alphaXiv.