Reproduce “Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tao Ji, Bin Guo, Yuanbin Wu, Qipeng Guo, Lixing Shen, Zhan Chen, Xipeng Qiu, Qi Zhang and 1 more
A framework termed MHA2MLA enables pre-trained large language models to adopt DeepSeek's Multi-Head Latent Attention architecture, significantly reducing Key-Value cache size by up to 96.87% with only marginal performance degradation. The method requires a small fraction of original pre-training data for fine-tuning, making efficient LLM inference accessible for existing models.
Key insights
- The "Head-wise 2-norm Contribution" strategy effectively identifies and preserves RoPE dimensions most critical for attention scores, outperforming frequency-based or uniform selection.
- Jointly factorizing key and value projection matrices via SVD ("SVD_joint") is superior to decoupled factorization, as it better maintains pre-trained knowledge by preserving key-value interdependencies.
- The architectural transformation to MLA complements existing KV cache quantization techniques, allowing for synergistic memory savings without compounding performance loss.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.