Reproduce “LatentLLM: Attention-Aware Joint Tensor Compression” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Toshiaki Koike-Akino, Xiangyu Chen, Jing Liu, Ye Wang, Wang, Matthew Brand
LatentLLM introduces a training-free framework to compress large language models and multi-modal models by converting Multi-Head Attention into a Multi-Head Latent Attention architecture. The method consistently achieves superior perplexity for LLMs and higher multi-modal reasoning accuracy for LMMs, demonstrating up to 40% parameter reduction while often improving performance over uncompressed baselines.
Key insights
- The square-root covariance of input activations serves as the optimal pre-conditioning matrix for activation-aware SVD, leading to improved low-rank approximation accuracy.
- A strategically chosen 'junction matrix' can effectively create block identity factors within low-rank decompositions, thereby enhancing parameter and computational efficiency without compromising approximation.
- Globally optimizing interdependent layers, such as joint Query-Key (QK) and Value-Output (VO) compression in attention or Up-Down (UD) projections in MLPs, yields better overall model performance than independent layer-wise compression.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.