Reproduce “Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan and 2 more

Tune-A-Video introduces a one-shot tuning approach to adapt pre-trained text-to-image diffusion models for text-to-video generation using only a single text-video pair. The method achieves high-quality, temporally consistent video generation and editing by inflating model architecture, employing efficient spatio-temporal attention, and utilizing DDIM inversion for structural guidance.

Key insights

  • Pre-trained text-to-image diffusion models implicitly capture motion dynamics and content consistency, which can be leveraged for video generation.
  • Sparse spatio-temporal attention and selective fine-tuning enable efficient adaptation of 2D image models to the video domain without extensive video data.
  • DDIM inversion effectively preserves the structural movement from an input video, allowing new content to be generated while maintaining original motion.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.