Reproduce “Photorealistic Video Generation with Diffusion Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang and 1 more

W.A.L.T (Window Attention Latent Transformer) is a transformer-based approach that demonstrates the first successful empirical joint training of image and video latent diffusion models using a pure transformer backbone. It achieves state-of-the-art performance across class-conditional image and video generation, frame prediction, and high-resolution text-to-video synthesis.

Key insights

  • Transformers, when equipped with efficient windowed attention mechanisms, can effectively replace U-Net architectures for video generation, demonstrating scalability benefits for this domain.
  • Joint training on both image and video datasets within a unified latent space significantly improves performance for both modalities, addressing challenges posed by video data scarcity.
  • Parameter-efficient conditioning techniques like AdaLN-LoRA contribute to maintaining model effectiveness while reducing computational overhead for incorporating diverse conditioning signals.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.