Reproduce “Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers” on OpenResearch

About this paper

Pengtao Chen, Xianfang Zeng, Maosen Zhao, Peng Ye, Mingzhu Shen, Wei Cheng, Gang Yu, Tao Chen

Researchers from Fudan University, StepFun, CUHK, and Imperial College London develop Sparse-vDiT, a framework that exploits content-invariant sparsity patterns in Video Diffusion Transformers to achieve up to 2.38× FLOP reduction and 1.85× inference speedup while maintaining superior generation quality compared to existing acceleration methods, discovering through systematic analysis that vDiT attention maps exhibit three recurring patterns (diagonal, multi-diagonal, vertical-stripe) plus redundant heads that remain stable across different input content, enabling an offline search algorithm to determine optimal sparse configurations for each attention head followed by specialized kernel implementations and head fusion optimizations that translate theoretical computational savings into practical hardware acceleration across state-of-the-art models including CogVideoX1.5, HunyuanVideo, and Wan2.1.

Key insights

• vDiT attention exhibits structured sparsity patterns that remain stable across different video content, with pattern distribution strongly correlated to architectural position (layer depth, head index) rather than input characteristics
• The content-invariance property enables offline optimization without runtime overhead, as the optimal sparse configuration can be determined once and applied to all subsequent inferences
• Balancing reconstruction error and sparsity through the loss function L_i = MSE(O_i - O_0) + λ × (1 - S_i) provides controllable quality-efficiency trade-offs via hyperparameters λ and ε

Summary by alphaXiv.