Reproduce “Veda: Scalable Video Diffusion via Distilled Sparse Attention” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shihao Han, Hao Yang, Xinting Hu, Xiaofeng Mei, Yi Jiang, Xiaojuan Qi
Veda reduces video diffusion Transformer attention work by predicting which hardware-sized token tiles to retain, using full-attention scores for training and head-specific tile layouts selected offline. A tile-skipping kernel applies the resulting masks. For Waver-T2V-12B at 720P and 241 frames, generation time falls from 19.4 to 3.8 minutes at 95% sparsity; quality retention varies by model and setting.
Key insights
- Video generation quality with sparse attention is largely dependent on the fidelity of the sparse mask in aligning with the underlying full attention structure, rather than just the sparsity level.
- Attention patterns exhibit high heterogeneity across different transformer heads and diffusion timesteps, indicating the need for adaptive, head-specific tiling configurations.
- Using richer, extrema-aware statistical representations (Average, Maximum, Minimum) for tile descriptors significantly improves the reconstruction of full attention structure.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.