Reproduce “LumosFlow: Motion-Guided Long Video Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiahao Chen, Hangjie Yuan, Yichen Qian, Jingyun Liang, Jiazheng Xing, Pengwei Liu, Weihua Chen, Fan Wang and 1 more
LumosFlow introduces a motion-guided hierarchical pipeline for long video generation, leveraging optical flow to ensure fluid motion and temporal consistency. The framework, developed by DAMO Academy, Renmin University of China, and Zhejiang University, achieves a 98.2% human preference rate and an FVD of 912.83, outperforming existing methods in generating high-quality, long-form videos.
Key insights
- Explicitly incorporating optical flow information into hierarchical video generation addresses a critical gap, enabling more fluid and consistent long videos.
- Decomposing the long video generation process into key frame generation, motion prediction, and frame refinement provides a modular and scalable architecture.
- Compressing optical flow data with a dedicated VAE (OF-VAE) and using linear flow initialization significantly aids in generating accurate and robust motion priors.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.