Reproduce “Unsupervised 3D Perception with 2D Vision-Language Distillation for Autonomous Driving” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Mahyar Najibi, Jingwei Ji, Yin Zhou, Charles R. Qi, Xinchen Yan, Scott Ettinger, Dragomir Anguelov

A framework called UP-VL enables unsupervised open-vocabulary 3D object detection and tracking for autonomous driving by distilling semantic knowledge from 2D vision-language models into 3D LiDAR data. It detects objects in all motion states and assigns semantic labels without relying on 3D human annotations, achieving 52.8 mAP for class-aware moving objects and 35.8 mAP for all-motion-states open-vocabulary detection on the Waymo Open Dataset.

Key insights

  • Fusing intrinsic LiDAR motion signals with rich semantic knowledge extracted from 2D vision-language models enables comprehensive unsupervised 3D perception across all object motion states.
  • Transferring pre-trained 2D vision-language features to 3D LiDAR points and distilling them into a 3D detector's latent space successfully imbues the model with open-vocabulary semantic understanding without 3D human labels.
  • Generating amodal 3D bounding boxes and tracklets by aggregating object shapes over time significantly enhances auto-label quality and subsequent detector performance in challenging autonomous driving scenarios.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.