Reproduce “Reconstruct, Inpaint, Test-Time Finetune: Dynamic Novel-view Synthesis from Monocular Videos” on OpenResearch

About this paper

Kaihua Chen, Tarasha Khurana, Deva Ramanan

Carnegie Mellon University researchers introduce CogNVS, a video diffusion model pipeline for dynamic novel-view synthesis from monocular videos, that integrates 3D reconstruction, structured inpainting, and test-time finetuning. The method achieves state-of-the-art photometric and generative performance across synthetic and real-world datasets, demonstrating robust zero-shot generalization and outperforming previous methods like Gen3C with a PSNR of 22.63 on Kubric-4D.

Key insights

• Reframing dynamic novel-view synthesis as a structured video inpainting problem allows leveraging powerful video diffusion models for generating 3D-consistent, novel content.
• Self-supervised training on readily available 2D videos, by generating reconstruction-based masks, effectively teaches a model to hallucinate previously unobserved scene parts in a geometrically plausible way, circumventing the need for expensive 4D datasets.
• The combination of robust pre-training and test-time finetuning provides a powerful mechanism for achieving both strong generalization to new domains (zero-shot) and high scene-specific fidelity in dynamic NVS.

Summary by alphaXiv.