Reproduce “VisionNVS: Self-Supervised Inpainting for Novel View Synthesis under the Virtual-Shift Paradigm” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hongbo Lu, Liang Yao, Chenghao He, Fan Liu, Wenlong Liao, Tao He, Pai Peng
Researchers from Shanghai Jiao Tong University and COWARobot Co. Ltd. developed VisionNVS, a camera-only framework that reformulates novel view synthesis for autonomous driving as a self-supervised inpainting task, bypassing the need for LiDAR sensors. The approach generated photorealistic and geometrically consistent views, achieving an FID of 11.98 for 1m shifts and 16.05 for 4m shifts on the nuScenes dataset, while offering three times faster inference compared to previous state-of-the-art generative methods.
Key insights
- Reframing novel view synthesis as an inpainting problem allows for self-supervised learning with pixel-perfect ground truth from raw, artifact-free images, directly addressing the supervision gap.
- High-fidelity and geometrically consistent NVS for autonomous driving is achievable using only camera inputs, removing the need for costly LiDAR sensors.
- Training the model with imperfectly stitched context from neighboring cameras, which naturally contains discrepancies, helps in learning robust feature correction for improved spatial consistency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.