Reproduce “4Real-Video: Learning Generalizable Photo-Realistic 4D Video Diffusion” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chaoyang Wang, Peiye Zhuang, Tuan Duc Ngo, Willi Menapace, Aliaksandr Siarohin, Michael Vasilkovsky, Ivan Skorokhodov, Sergey Tulyakov and 2 more
The 4Real-Video framework from Snap Inc., Umass Amherst, and KAUST generates photo-realistic 4D videos, defined as a grid of frames aligned across time and viewpoint. It achieves superior multi-view and temporal consistency and significantly faster inference, generating an 8x8 grid in roughly one minute compared to hours for prior methods, through a novel two-stream architecture and adaptive synchronization.
Key insights
- Parallel processing of multi-view and temporal information with adaptive synchronization is more effective for maintaining 4D consistency than sequential or rigid merging approaches.
- Leveraging large-scale 2D video data for base model training, combined with targeted fine-tuning on small 4D datasets, enables generalization to photorealistic real-world scenarios.
- The proposed architecture can infer viewpoints and motion from simple conditional video inputs, simplifying the generation process without explicit camera conditioning modules.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.