Reproduce “Novel View Synthesis with Diffusion Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Daniel Watson, William Chan, Ricardo Martin-Brualla, Jonathan Ho, Andrea Tagliasacchi, Mohammad Norouzi
Google Research developed 3DiM, a geometry-free diffusion model for novel view synthesis that generates sharp, high-fidelity views while maintaining approximate 3D consistency. The model achieves superior results on the SRN ShapeNet dataset by employing stochastic conditioning and a specialized X-UNet architecture.
Key insights
- Stochastic conditioning, by randomly selecting conditioning views from previously generated views, effectively promotes 3D consistency across autoregressively generated novel views.
- The X-UNet architecture, with shared weights, cross-attention layers, and unified conditioning, is essential for processing multiple views symmetrically and maintaining consistency.
- Traditional image quality metrics like PSNR/SSIM and FID often fail to capture 3D consistency, necessitating a specialized '3D consistency scoring' method for geometry-free view synthesis models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.