Reproduce “Tucker Attention: A generalization of approximate attention mechanisms” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Timon Klein, Jonas Kusch, Sebastian Sager, Stefan Schnake, Steffen Schotthöfer
Tucker Attention introduces a tensor-based reformulation of multi-head attention, applying low-rank Tucker decomposition to generalize and unify existing parameter-efficient methods like MQA, GQA, and MLA. This approach achieves an order of magnitude reduction in attention parameters and significantly smaller KV cache sizes in models such as ViT, GPT-2, and LLaMA3-1B, while maintaining competitive performance and full compatibility with current transformer ecosystems.
Key insights
- MHA and existing approximate attention mechanisms (MQA, GQA, MLA) are special cases of Tucker Attention, providing a unified theoretical framework for their underlying low-rank behavior.
- Analysis of trained models reveals significant unexploited low-rank structures across all tensor modes (head, query/output, key/value), offering further compression potential.
- A direct tensor decomposition approach allows for superior parameter efficiency compared to methods that compress individual matrices or only certain modes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.