Reproduce “Sparse Multiview Open-Vocabulary 3D Detection” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Olivier Moliner, Viktor Larsson, Kalle Åström
Sparse Multiview Open-Vocabulary 3D Detection (SMOV3D) is a training-free pipeline that performs 3D object detection from sparse RGB images by orchestrating off-the-shelf 2D foundation models and multi-view consistency optimization. This method achieves competitive performance against state-of-the-art 3D detectors while outperforming them in sparse-view and long-tail conditions on benchmarks like ScanNet and Replica.
Key insights
- Powerful generalization capabilities of 2D vision-language foundation models can be directly and effectively transferred to complex 3D perception tasks without requiring any 3D-specific training.
- Robust 3D inference from sparse monocular estimates necessitates multi-view consistency, which is optimally achieved by combining both photometric and semantic (CLIP feature) cues.
- A training-free approach, by not inheriting biases from 3D training datasets, can exhibit strong zero-shot generalization, particularly excelling on diverse and long-tail object categories.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.