Reproduce “3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ji Hou, Angela Dai, Matthias Nießner
A new deep learning architecture from the Technical University of Munich, 3D-SIS, performs 3D semantic instance segmentation of RGB-D scans by jointly learning features from high-resolution 2D RGB data and 3D geometry. This method significantly improved mean Average Precision (mAP) by over 13.5 points on the challenging ScanNetV2 real-world dataset, establishing a new state of the art.
Key insights
- Joint learning from high-resolution 2D RGB features and 3D geometric data consistently and substantially improves performance for 3D instance segmentation.
- Back-projecting 2D CNN features onto the 3D voxel grid using precise camera poses effectively integrates high-resolution visual cues into a 3D volumetric representation.
- Maintaining high spatial resolution throughout the 3D mask prediction backbone is critical for obtaining accurate per-voxel instance predictions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.