Reproduce “SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xinyuan Chen, Yaohui Wang, Lingjun Zhang, Shaobin Zhuang, Xin Ma, Jiashuo Yu, Yali Wang, Dahua Lin and 2 more
The SEINE model introduces a short-to-long video diffusion framework for generating creative transitions between diverse scenes and predicting extended video sequences. It extends a pre-trained text-to-video model, quantitatively surpassing traditional morphing and diffusion-based baselines in human evaluations and objective metrics for transition quality, and maintains visual coherence in long-form video generation.
Key insights
- A random mask module can effectively enable a diffusion model to control which frames are generated versus conditioned upon, facilitating versatile short-to-long video generation tasks.
- Training a diffusion model on partial, masked video inputs (visible and unseen frames) allows it to learn robust mechanisms for bridging semantic and temporal gaps between diverse scenes.
- Extending pre-trained text-to-video diffusion models, rather than training from scratch, provides a strong foundation for advanced generative tasks like story-level video creation and complex transitions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.