Reproduce “OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhening Huang, Xiaoyang Wu, Xi Chen, Hengshuang Zhao, Lei Zhu, Joan Lasenby

OpenIns3D introduces a 3D open-vocabulary instance segmentation framework that operates exclusively on 3D point cloud inputs, bypassing the need for aligned 2D images. The system achieved a 60.8% Top-1 accuracy on ScanNetv2 for open-vocabulary recognition and an AP50 of 37.0% on novel categories for instance segmentation on S3DIS.

Key insights

  • Synthetically generated multi-view 2D images can effectively bridge the domain gap, enabling 2D vision-language models to process 3D scene information without external 2D inputs.
  • Decoupling 3D mask proposal from 2D semantic classification allows the framework to readily integrate and leverage advancements in 2D open-vocabulary detectors.
  • Robust mask scoring and filtering mechanisms are crucial for refining class-agnostic 3D proposals, significantly enhancing the quality of instances passed to the 2D classification stage.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.