Reproduce “OrbitNVS: Harnessing Video Diffusion Priors for Novel View Synthesis” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jinglin Liang, Zijian Zhou, Rui Huang, Shuangping Huang, Yichen Gong
OrbitNVS reformulates Novel View Synthesis as an orbital video generation task, adapting a video diffusion model with a Camera Adapter, Normal Map Generation Branch, and Pixel-Space Post-Training. This method improves view synthesis quality, achieving +2.9 dB PSNR on GSO and +2.4 dB PSNR on OmniObject3D in single-view NVS compared to prior state-of-the-art methods.
Key insights
- Repurposing large-scale pre-trained video diffusion models for Novel View Synthesis by framing it as an orbital video generation task, leveraging their learned temporal and visual priors.
- Accurate camera pose conditioning, achieved through a dedicated Camera Adapter utilizing Plücker coordinates and AdaLN, is critical for precise and controllable view synthesis.
- Explicit geometric guidance via an auxiliary Normal Map Generation Branch significantly enhances 3D consistency and detailed shape generation in synthesized views.
- Pixel-space post-training effectively recovers high-frequency visual details lost during latent space processing, leading to sharper and more photorealistic results.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.