Reproduce “HoloDrive: Holistic 2D-3D Multi-Modal Street Scene Generation for Autonomous Driving” on OpenResearch

About this paper

Zehuan Wu, Jingcheng Ni, Xiaodong Wang, Yuxin Guo, Rui Chen, Lewei Lu, Jifeng Dai, Yuwen Xiong

HoloDrive, developed by researchers from Sensetime Research, Shanghai AI Lab, Tsinghua University, and Peking University, introduces a framework for generating holistic and consistent 2D-3D multi-modal street scenes, including multi-view camera images and corresponding LiDAR point clouds. The approach achieves state-of-the-art performance in both single-frame and video generation benchmarks on the NuScenes dataset, outperforming existing single-modality and preliminary joint-modality methods.

Key insights

• Effective and consistent joint generation of heterogeneous 2D camera images and 3D LiDAR point clouds is achievable through explicit bi-directional feature transformation and depth supervision.
• Integrating 3D spatial awareness into 2D generative models via depth prediction and 3D-to-2D feature projection significantly enhances the realism and structural fidelity of generated images and their alignment with 3D data.
• A progressive multi-task training strategy, incorporating conditional and joint dropping policies, is crucial for training a unified temporal 2D-3D world model, improving cross-modal learning and overall performance.

Summary by alphaXiv.