Reproduce “Long-Term Photometric Consistent Novel View Synthesis with Diffusion Models” on OpenResearch

About this paper

Jason J. Yu, Fereshteh Forghani, Konstantinos G. Derpanis, Marcus A. Brubaker

Researchers at York University and the Vector Institute for AI developed an autoregressive conditional diffusion model that generates long-term photometrically and geometrically consistent novel view sequences from a single input image and a camera trajectory. This model outperforms existing baselines in image quality and consistency, particularly for novel camera paths, and introduces a new metric, Thresholded Symmetric Epipolar Distance (TSED), for evaluating geometric consistency.

Key insights

• Autoregressive latent diffusion models can effectively handle the high uncertainty of view extrapolation by generating diverse yet plausible content for unobserved scene regions.
• Explicitly encoding camera ray representations and using a two-stream cross-attention mechanism in the diffusion model significantly enhances long-term photometric and geometric consistency in generated view sequences.
• Evaluating view extrapolation methods requires specialized metrics like TSED, which directly assess geometric consistency, as traditional image quality metrics alone do not fully capture the fidelity of novel view generation.

Summary by alphaXiv.