Reproduce “Enhancing Novel View Synthesis via Geometry Grounded Set Diffusion” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Farhad G. Zanjani, Hong Cai, Amirhossein Habibian
SetDiff enhances rendered novel-view images from 3D Gaussian Splatting scenes for driving simulation. It conditions a diffusion model on multiple reference and target views, coordinate maps, and camera-ray geometry, aggregating information across views. In Para-Lane double-lane extrapolation, it records PSNR 19.47, SSIM 0.564, and LPIPS 0.233, outperforming a same-backbone baseline; these image gains do not establish accurate recovery of unseen scene content.
Key insights
- Explicit 3D geometric conditioning, combining per-pixel world coordinate maps and Plücker ray camera pose embeddings, significantly enhances the ability of diffusion models to reason about 3D structures and correct NVS artifacts.
- A "Set Mixer" architecture, which performs multi-head self-attention over a unified set of tokens from multiple reference and target views, enables robust and scalable multi-view information aggregation.
- Augmenting the latent diffusion objective with a pixel-space reconstruction loss, allowing for joint fine-tuning of the VAE decoder, improves both structural consistency and perceptual fidelity of the enhanced images.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.