Reproduce “ERC-SVD: Error-Controlled SVD for Large Language Model Compression” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Haolei Bai, Siyong Jian, Tuo Liang, Yu Yin, Huan Wang

ERC-SVD, an error-controlled Singular Value Decomposition method, compresses large language models by integrating residual compensation for truncation loss and partial-layer compression to mitigate error propagation. The method demonstrates improved perplexity and zero-shot accuracy across various LLMs and scales, outperforming existing SVD-based methods and enabling faster inference with compatibility for 8-bit quantization.

Key insights

  • Explicitly leveraging the residual matrix generated during SVD truncation significantly reduces approximation error, leading to a more accurate low-rank representation.
  • Error propagation in LLMs is highly detrimental to performance; compressing only the last few layers effectively mitigates this issue by preserving early, critical computations.
  • The final-layer error strongly correlates with overall zero-shot accuracy, providing a reliable metric for optimizing layer-wise compression strategies.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.