Reproduce “VecAttention: Vector-wise Sparse Attention for Accelerating Long Context Inference” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Anmin Liu, Ruixuan Yang, Huiqiang Jiang, Bin Lin, Minmin Sun, Yong Li, Chen Zhang, Tao Xie

VecAttention is a framework that leverages empirically observed fine-grained vertical-vector sparsity to accelerate long-context inference in Transformer-based video models. It achieves up to 2.65x speedup over full attention and 1.83x over prior sparse methods for attention computation while preserving model accuracy.

Key insights

  • Attention maps in Transformer-based video models exhibit an intrinsic fine-grained vertical-vector sparse pattern, which is a more accurate representation of salient interactions than coarse-grained patterns.
  • Co-optimizing algorithms with hardware considerations (e.g., minS filtering, TilingSelect, cross-tile minS optimization) can effectively mitigate the overheads of fine-grained sparsity selection.
  • Per-head dynamic programming to determine optimal filter ratios allows VecAttention to adapt to varying sparsity characteristics across different attention heads, improving overall accuracy under sparsity constraints.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.