Reproduce “FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xingxing Zuo, Pouya Samangouei, Yunwen Zhou, Yan Di, Mingyang Li
FMGS integrates 3D Gaussian Splatting with Multi-resolution Hash Encoding to enable real-time, open-vocabulary 3D scene understanding. The approach achieves 93.2% accuracy in object detection, outperforming LERF by 10.2 percentage points, and renders feature maps over 850 times faster.
Key insights
- Using Multi-resolution Hash Encoding (MHE) for semantic features significantly reduces memory consumption and parameter count compared to directly attaching features to each Gaussian.
- A hybrid multi-scale CLIP feature for supervision, combined with a DINO-based pixel-alignment loss, is crucial for learning robust, pixel-accurate, and multi-view consistent 3D semantic features.
- 3D Gaussian Splatting's inherent efficiency makes it a superior backbone for real-time 3D semantic understanding and open-vocabulary querying compared to NeRF-based approaches.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.