Reproduce “ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Image” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kyle Sargent, Zizhang Li, Tanmay Shah, Charles Herrmann, Hong-Xing Yu, Yunzhi Zhang, Eric Ryan Chan, Dmitry Lagun and 3 more
ZeroNVS extends diffusion-guided novel-view synthesis from isolated objects to unconstrained real-world scenes, generating 360-degree-consistent views from a single image. Its six-degree-of-freedom, viewer-centric camera normalization and SDS anchoring improve perceptual quality and background diversity, achieving LPIPS scores of 0.380 on DTU and 0.625 on Mip-NeRF 360, while anchoring was preferred in 80% of user-study comparisons.
Key insights
- Relative camera pose is insufficient when training scenes have arbitrary coordinate scales; estimating scale from the input view produces more stable conditioning than aggregating information from all views.
- Diffusion-guided 3D optimization can become mode-seeking, often yielding gray or repetitive unseen backgrounds. Anchoring the distillation with independently sampled views broadens the generated scene hypotheses without strongly altering foreground content.
- Perceptual metrics are important for wide-baseline synthesis: PixelNeRF can obtain better PSNR or SSIM on Mip-NeRF 360 while producing worse LPIPS, indicating lower visual plausibility.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.