Reproduce “ROGR: Relightable 3D Objects using Generative Relighting” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiapeng Tang, Matthew Levine, Dor Verbin, Stephan J. Garbin, Matthias Nießner, Ricardo Martin Brualla, Pratul P. Srinivasan, Philipp Henzler
ROGR enables object-centric relighting by distilling jointly relit multi-view examples into one lighting-conditioned NeRF. It trains on 111 environment maps, using general and reflected-direction specular conditioning to avoid per-light NeRF optimization at inference. On TensoIR, it averages 30.74 PSNR, 0.950 SSIM, and 0.069 LPIPS-VGG, outperforming IllumiNeRF; these benchmark averages do not establish generalization to maps held out from distillation.
Key insights
- Leveraging a generative multi-view diffusion model to create a view-consistent, multi-illumination dataset can effectively overcome the data and consistency limitations of training relightable 3D representations.
- A dual-branch NeRF conditioning scheme, separating general lighting from explicit specular reflection, allows for robust handling of both diffuse and high-frequency material properties under novel illumination.
- Learning compact embeddings from environment maps enables a NeRF to generalize to arbitrary unseen lighting conditions at inference time, achieving feed-forward relighting without per-illumination optimization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.