Reproduce “RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Titas Anciukevičius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy J. Mitra, Paul Guerrero

RenderDiffusion makes each image-denoising step construct a triplane-based 3D scene and render it as an image, enabling reconstruction and novel-view rendering from monocular 2D supervision. Training assumes camera intrinsics and extrinsics. On ShapeNet, its average reconstruction PSNR is 26.1, versus 24.1 for single-view-trained EG3D; multi-view-trained PixelNeRF scores 28.9, so this is not a matched comparison.

Key insights

  • Deeply embedding a compact triplane representation and a volumetric renderer into a 2D image denoiser provides a strong 3D inductive bias, enabling 3D reasoning from 2D supervision.
  • Diffusion models can effectively learn to generate and reconstruct consistent 3D scenes by operating on 2D images, bypassing the need for expensive and scarce 3D datasets.
  • The diffusion sampling process can be adapted for monocular 3D reconstruction, offering a trade-off parameter ($t_r$) to balance fidelity to the input with generalization to out-of-distribution images.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.