Reproduce “DiffDreamer: Towards Consistent Unsupervised Single-view Scene Extrapolation with Conditional Diffusion Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shengqu Cai, Eric Ryan Chan, Songyou Peng, Mohamad Shahbazi, Anton Obukhov, Luc Van Gool, Gordon Wetzstein
DiffDreamer introduces the first diffusion-based framework for single-view scene extrapolation, generating consistent novel views for long camera trajectories. The model demonstrates improved spatio-temporal and 3D consistency over prior methods, achieving higher quantitative metrics like PSNR and COLMAP reconstruction points when fusing generated frames into a 3D scene.
Key insights
- Conditional diffusion models' iterative refinement process and high-fidelity generation capabilities are well-suited for the joint inpainting and 3D refinement task of scene extrapolation.
- Maintaining long-range spatio-temporal and 3D consistency requires explicit conditioning on both historical ("anchored") and future-projected ("virtual lookahead") scene information during inference.
- Unsupervised training on single images can effectively teach a diffusion model to synthesize consistent novel views by creating synthetic corrupted training pairs through forward camera motion.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.