Reproduce “OVGaussian: Generalizable 3D Gaussian Segmentation with Open Vocabularies” on OpenResearch

About this paper

Runnan Chen, Xiangyu Sun, Zhaoqing Wang, Youquan Liu, Jiepeng Wang, Lingdong Kong, Jiankang Deng, Mingming Gong and 3 more

The OVGaussian framework extends open-vocabulary semantic segmentation to 3D Gaussian representations, enabling generalization across diverse and unseen scenes. It achieves state-of-the-art performance, with a 43.84% mIoU in 3D cross-scene segmentation and 18.93% mIoU in 3D cross-domain segmentation, by leveraging cross-modal consistency learning and a new dataset, SegGaussian.

Key insights

• Learning view-independent semantic vectors for 3D Gaussians, facilitated by a multi-granularity 3D neural network, enables consistent semantic representations across varying viewpoints and scenes.
• Aligning semantic information across 3D Gaussians, 2D visual features, and text embeddings via cross-modal consistency learning is effective for achieving robust open-vocabulary segmentation and generalization.
• A Voxel-to-Point Adapter within the 3D neural network is crucial for integrating global and local spatial context, refining semantic vectors for individual Gaussians.

Summary by alphaXiv.