Reproduce “NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusion” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jiatao Gu, Alex Trevithick, Kai-En Lin, Josh Susskind, Christian Theobalt, Lingjie Liu, Ravi Ramamoorthi

NerfDiff turns one object photograph into views from other angles. It uses diffusion to propose unseen details, then alternates guidance from a shared NeRF with updates to that 3D scene representation. On ShapeNet Cars, NGD yields FID 18.09 and LPIPS 0.082, versus 35.88 and 0.105 for direct distillation; its PSNR, 23.51, is below 23.81 without finetuning in this ablation.

Key insights

  • Combining a 3D-aware conditional diffusion model with a NeRF, and using a NeRF-guided distillation process, allows for robust hallucination of unseen regions while maintaining geometric consistency.
  • A camera-aligned triplane representation for the NeRF efficiently encodes 3D scenes and is effective for single-view inputs by allocating capacity along depth where uncertainty is high.
  • The NeRF-guided distillation (NGD) algorithm provides a more stable and faster converging optimization process compared to Score Distillation Sampling (SDS) for refining single-image NeRFs.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.