Reproduce “VideoMV: Consistent Multi-View Generation Based on Large Video Generative Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Qi Zuo, Xiaodong Gu, Lingteng Qiu, Yuan Dong, Zhengyi Zhao, Weihao Yuan, Rui Peng, Siyu Zhu and 3 more
VideoMV, developed by researchers from Alibaba Group's Institute for Intelligent Computing and collaborators, presents a framework for consistent multi-view image generation by leveraging pre-trained video generative models. The method fine-tunes these models and integrates a novel 3D-aware denoising sampling strategy, drastically reducing training time to 4 GPU hours while producing high-quality, consistent 24-view images that surpass prior techniques.
Key insights
- Pre-trained large video generative models inherently learn strong temporal consistency, which can be directly transferred and effectively leveraged for consistent multi-view image generation.
- Integrating explicit 3D information (via 3D Gaussians and a feed-forward reconstruction network) into the diffusion model's denoising loop can significantly enhance multi-view consistency and 3D structure coherence.
- Combining fine-tuning on rendered 3D-centric videos with general 2D image data maintains both multi-view consistency and broad generalizability for diverse prompts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.