Reproduce “PSA: Pyramid Sparse Attention for Efficient Video Understanding and Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiaolong Li, Youping Gu, Xi Lin, Weijie Wang, Bohan Zhuang
Pyramid Sparse Attention (PSA) from Zhejiang University's ZIP Lab introduces a hierarchical sparse attention mechanism for efficient video understanding and generation, utilizing multi-level pooled key-value representations to adaptively allocate computational resources. This approach mitigates information loss inherent in binary sparse masks, achieving superior perceptual quality in video generation at high sparsity and up to a 10x speedup in hardware-optimized implementations.
Key insights
- Aggregating adjacent key tokens through multi-level pooling is effective due to their strong local similarity in visual sequences, allowing for a continuous degradation of precision rather than abrupt pruning.
- Dynamic, threshold-based multi-level masking, combined with Hilbert curve permutation for improved locality, offers a more adaptive and effective computation allocation strategy than fixed binary or quantile-based rules.
- Decoupling logical attention block sizes from hardware execution tile sizes is crucial for maintaining high GPU throughput and avoiding warp divergence when implementing flexible, heterogeneous sparse attention patterns.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.