Reproduce “ARA: Adaptive Rank Allocation for Efficient Large Language Model SVD Compression” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Lin Xv, Jingsheng Gao, Xian Gao, Ting Liu, Yuzhuo Fu

Researchers at Shanghai Jiao Tong University developed Adaptive Rank Allocation (ARA), a method for Singular Value Decomposition (SVD) compression of Large Language Models (LLMs) that optimally distributes low-rank approximations across model layers. This approach consistently outperforms existing SVD compression baselines, achieving a 9.72 percentage point improvement in average zero-shot accuracy for LLaMA2-7B at 80% compression while remaining compatible with other techniques like quantization.

Key insights

  • An effective mask design for SVD-based rank allocation must ensure both monotonic retention of singular values (prioritizing larger ones) and global influence of mask parameters on the singular value spectrum.
  • The behavior of SVD's gain function at a compression ratio of 1 is non-smooth; dynamically allowing modules to revert to full-rank when decomposition is suboptimal is crucial for globally optimal compression.
  • Optimally allocating singular value ranks across different LLM layers can significantly improve performance retention compared to uniform compression or previous adaptive strategies, even at high compression rates.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.