Reproduce “4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Heng Yu, Chaoyang Wang, Peiye Zhuang, Willi Menapace, Aliaksandr Siarohin, Junli Cao, Laszlo A Jeni, Sergey Tulyakov and 1 more
The "4Real" research introduces a pipeline for photorealistic text-to-4D scene generation by utilizing video diffusion models trained on diverse real-world video datasets, moving away from multi-view models fine-tuned on synthetic data. This approach generates dynamic scenes featuring animated objects within detailed environments, exhibiting improved visual quality and computational efficiency.
Key insights
- Leveraging real-world video diffusion models as generative priors, rather than multi-view models trained on synthetic data, is crucial for achieving photorealistic and diverse 4D scene generation.
- Multi-view inconsistencies in "freeze-time" videos can be effectively modeled as per-frame deformations during 3D Gaussian Splat reconstruction, allowing robust canonical 3D representation from noisy inputs.
- Utilizing a low-resolution, pixel-space video diffusion model for Score Distillation Sampling drastically improves computational efficiency, reducing generation time from over 10 hours to approximately 1.5 hours.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.