Reproduce “CLIP-GS: CLIP-Informed Gaussian Splatting for View-Consistent 3D Indoor Semantic Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Guibiao Liao, Jiankun Li, Zhenyu Bao, Xiaoqing Ye, Qing Li, Kanglin Liu
Researchers from Peking University, Pengcheng Laboratory, and Baidu Inc. developed CLIP-GS, an extension to 3D Gaussian Splatting that achieves real-time, view-consistent 3D indoor semantic understanding. It integrates compact semantic attributes and multi-level consistency regularization, significantly boosting segmentation accuracy by over 21% mIoU on ScanNet while delivering semantic rendering speeds exceeding 100 frames per second.
Key insights
- Embedding low-dimensional, compact semantic attributes directly onto 3D Gaussians is critical for enabling real-time semantic rendering without compromising quality.
- Enforcing semantic consistency at multiple levels—across associated 2D views and among linked 3D Gaussian primitives—is essential for achieving robust, view-invariant 3D semantic understanding.
- Regulating the densification of 3D Gaussians throughout training, starting with lower resolutions and stricter thresholds, optimizes the resulting semantic radiance field for both efficiency and accuracy.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.