Reproduce “OnlinePG: Online Open-Vocabulary Panoptic Mapping with 3D Gaussian Splatting” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hongjia Zhai, Qi Zhang, Xiaokun Pan, Xiyu Zhang, Yitong Dong, Huaqi Zhang, Dan Xu, Guofeng Zhang
OnlinePG introduces a system for real-time open-vocabulary panoptic mapping, leveraging 3D Gaussian Splatting and a local-to-global paradigm to create consistent 3D instance maps from noisy 2D vision-language model outputs. It achieves approximately 13.8 mIoU and 11.5 mAcc improvement over prior online methods on ScanNetV2 and operates at 10-18 frames per second.
Key insights
- Integrating 3D Gaussian Splatting with fine-grained, grid-based spatial attributes offers an efficient and scalable way to represent and incrementally update both geometric and rich semantic/instance information in real-time.
- A local-to-global processing strategy, coupled with multi-cue segment clustering, is effective in consolidating noisy 2D VLM predictions into coherent and consistent 3D panoptic instances for online applications.
- Bidirectional bipartite matching provides a robust mechanism for fusing local instance maps into the global map, ensuring accurate correspondences and preventing feature drift and inconsistencies over time.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.