Reproduce “LangSplat: 3D Language Gaussian Splatting” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, Hanspeter Pfister
LangSplat, developed by researchers from Tsinghua and Harvard Universities, introduces a method for creating precise and efficient 3D language fields by integrating 3D Gaussian Splatting with hierarchical language features from SAM and CLIP. The approach enables real-time open-vocabulary querying in 3D, achieving a 199x speedup over previous state-of-the-art while improving 3D localization accuracy to 84.3% and semantic segmentation mIoU to 93.4% on respective benchmarks.
Key insights
- Leveraging SAM's hierarchical segmentation capabilities provides precise, boundary-aware features at multiple semantic levels (subpart, part, whole), effectively resolving point ambiguity in 3D language grounding.
- Representing 3D language fields with 3D Gaussian Splatting (3D-GS) enables real-time querying and rendering speeds, overcoming the computational bottlenecks of NeRF-based methods.
- A scene-wise language autoencoder efficiently compresses high-dimensional CLIP features into a low-dimensional latent space, making the explicit storage of language embeddings feasible for millions of 3D Gaussians without prohibitive memory consumption.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.