Reproduce “Weakly-Supervised 3D Visual Grounding based on Visual Language Alignment” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xiaoxu Xu, Yitian Yuan, Qiudan Zhang, Wenhui Wu, Zequn Jie, Lin Ma, Xu Wang

Researchers from Shenzhen University and Meituan Inc. developed 3D-VLA, a weakly supervised method for 3D visual grounding that leverages 2D images and pre-trained vision-language models to implicitly align text queries with 3D point clouds. This approach establishes state-of-the-art performance in weakly supervised 3D visual grounding, outperforming prior methods on ScanRefer by over 5% on [email protected] and showing competitive results against several fully supervised baselines.

Key insights

  • Pre-trained 2D vision-language models can be effectively leveraged to transfer robust semantic alignment capabilities from the 2D domain to 3D visual grounding tasks.
  • Utilizing 2D images, connected through geometric camera calibration, provides a practical and effective means to implicitly establish text-to-3D correspondences without direct 3D bounding box supervision.
  • Task-specific adaptation and intelligent filtering mechanisms, such as multi-modal adapters and category-oriented proposal filtering, are essential for improving the accuracy and robustness of weakly supervised 3D grounding models.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.