Reproduce “Layer-wise dynamic rank for compressing large language models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhendong Mi, Bian Sun, Grace Li Zhang, Shaoyi Huang

The D-Rank framework compresses Large Language Models by dynamically allocating SVD ranks across layers and within attention mechanisms based on an effective rank metric. This approach achieves lower perplexity and higher zero-shot accuracy, while significantly increasing inference throughput compared to uniform SVD compression methods.

Key insights

  • Information content within LLM weight matrices is highly heterogeneous, with a 'U-shaped' distribution across layers and higher density in attention W_V matrices compared to W_Q and W_K.
  • The 'effective rank' metric provides a principled and effective way to measure information density, enabling optimal, adaptive rank allocation for SVD compression.
  • Uniform SVD compression is suboptimal; a nuanced, architecture-aware approach with dynamic rank allocation significantly improves performance preservation and throughput for LLMs.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.