Reproduce “SegSplat: Feed-forward Gaussian Splatting and Open-Set Semantic Segmentation” on OpenResearch

About this paper

Peter Siegel, Federico Tombari, Marc Pollefeys, Daniel Barath

SegSplat introduces a purely feed-forward framework for generating 3D Gaussian Splatting representations that integrate open-set semantic features, developed by researchers from ETH Zürich, Google, and Microsoft. The method achieves 61.4% mIoU on the 3D-OVS dataset and generates semantically rich 3D scenes in under 0.2 seconds after initial 2D feature extraction, crucially without requiring per-scene semantic optimization, all while preserving geometric fidelity.

Key insights

• High-quality 2D open-set semantic information from foundation models can be effectively and efficiently integrated into 3D Gaussian Splatting without requiring any scene-specific semantic training or optimization.
• By compressing CLIP features into a shared semantic feature bank and assigning discrete one-hot indices to Gaussians, semantic information can be stored and recovered with minimal overhead, preserving efficiency.
• A unified feed-forward pipeline can deliver both high-fidelity geometric and appearance reconstruction and robust open-set semantic understanding, enabling practical applications in real-time scenarios.

Summary by alphaXiv.