Reproduce “Latte: Latent Diffusion Transformer for Video Generation” on OpenResearch

About this paper

Xin Ma, Yaohui Wang, Xinyuan Chen, Gengyun Jia, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, Yu Qiao

Latte introduces a Latent Diffusion Transformer, demonstrating that Transformer-based architectures can effectively serve as denoising backbones for high-quality video generation. The model achieves new state-of-the-art performance across multiple benchmarks and provides empirically derived design principles for building efficient spatio-temporal Transformers within diffusion models.

Key insights

• Transformer-based architectures are highly effective for latent diffusion video generation, consistently outperforming traditional CNN-based U-Nets and establishing a new architectural paradigm for video synthesis.
• Optimal design choices for video Transformers include using interleaved spatial and temporal attention blocks, uniform frame patch embedding, and Scalable Adaptive Layer Normalization (S-AdaLN) for efficient conditioning injection.
• A novel image-video joint training strategy, which leverages a 'Temporal Mask' to handle static images while training on videos, significantly improves video generation quality by benefiting from diverse data without compromising temporal coherence.

Summary by alphaXiv.