Reproduce “UltraLiDAR: Learning Compact Representations for LiDAR Completion and Generation” on OpenResearch

About this paper

Yuwen Xiong, Wei-Chiu Ma, Jingkang Wang, Raquel Urtasun

UltraLiDAR introduces a framework for learning compact, discrete representations of 3D LiDAR point clouds, which enables robust sparse-to-dense LiDAR completion and the generation of highly realistic, diverse, and controllable driving scenes. This approach enhances 3D perception performance and provides a scalable solution for augmenting autonomous vehicle datasets.

Key insights

• Discrete representation learning, previously successful in 2D image synthesis, can be effectively adapted to unstructured 3D LiDAR data by processing voxelized point clouds as BEV images within a VQ-VAE framework.
• Jointly optimizing a sparse encoder with a dense codebook allows for robust sparse-to-dense LiDAR completion, effectively bridging the performance gap between low-cost and high-density LiDAR sensors.
• LiDAR generation can be reframed as code map generation, where a generative Transformer can produce highly realistic and diverse scenes by iteratively predicting discrete codes, even supporting controllable manipulation like object insertion/removal.

Summary by alphaXiv.