Reproduce “Lumiere: A Space-Time Diffusion Model for Video Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur and 9 more
Lumiere, a Space-Time Diffusion Model developed by Google Research, synthesizes videos with realistic and coherent motion by employing a Space-Time U-Net that processes the full temporal duration in a single pass. User studies confirmed its superior video quality, motion, and text alignment compared to existing state-of-the-art models in text-to-video and image-to-video generation tasks.
Key insights
- Directly modeling the full temporal duration of a video via a single-pass Space-Time U-Net significantly improves global temporal coherence and motion realism compared to cascaded generation approaches.
- Processing videos in a multi-scale space-time representation, with combined spatial and temporal down/up-sampling, makes full-length video generation computationally tractable.
- Leveraging powerful pre-trained text-to-image priors by only fine-tuning newly added temporal parameters in the STUNet effectively adapts the model for robust video generation while inheriting strong visual quality.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.