Reproduce “MagicDrive3D: Controllable 3D Generation for Any-View Rendering in Street Scenes” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ruiyuan Gao, Kai Chen, Zhihao Li, Lanqing Hong, Zhenguo Li, Qiang Xu
MagicDrive3D introduces a novel framework for controllable 3D street scene generation that supports any-view rendering by integrating a multi-view video generation model with an enhanced 3D Gaussian Splatting pipeline. This approach achieves higher quality and controllable 3D scene generation from sparse driving datasets like nuScenes, demonstrated by improved FID scores for novel views and enhanced performance in BEV segmentation tasks.
Key insights
- Leveraging 2D controllable video generation to produce 'reconstruction-friendly' multi-view sequences is an effective strategy for overcoming data scarcity and inconsistency challenges in direct 3D street scene generation.
- Introducing local offsets to 3D Gaussians (Fault-Tolerant GS) and optimizing depth priors and appearance across views significantly improves the robustness and quality of 3D reconstruction from imperfect synthetic inputs.
- Precisely embedding ego-trajectory information into the video generation process is crucial for producing geometrically consistent 2D views suitable for subsequent 3D reconstruction.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.