Reproduce “GaussianCross: Cross-modal Self-supervised 3D Representation Learning via Gaussian Splatting” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lei Yao, Yi Wang, Yi Zhang, Moyun Liu, Lap-Pui Chau
Researchers at Hong Kong Polytechnic University and Huazhong University of Science and Technology present GaussianCross, a cross-modal self-supervised architecture for 3D representation learning that leverages feed-forward 3D Gaussian Splatting. The model achieves state-of-the-art performance in 3D semantic segmentation with 76.0% mIoU on ScanNet and 3D instance segmentation with 62.7% AP50 on ScanNet, exhibiting high data efficiency in low-data regimes.
Key insights
- 3D Gaussian Splatting can be effectively adapted from scene-specific optimization to a generalizable feed-forward framework suitable for self-supervised pre-training of 3D representations.
- Jointly capturing photometric appearance, geometric structure, and high-level semantic cues through multi-target rendering and distillation provides richer supervisory signals, effectively mitigating model collapse.
- Semantic-aware knowledge from powerful 2D Visual Foundation Models can be distilled into 3D point cloud features in a label-free manner, significantly reducing annotation dependency for 3D understanding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.