Reproduce “VGGT-Segmentor: Geometry-Enhanced Cross-View Segmentation” on OpenResearch

About this paper

Yulu Gao, Bohao Zhang, Zongheng Tang, Jitong Liao, Wenjun Wu, Si Liu

VGGT-Segmentor (VGGT-S), developed by researchers at Beihang University, improves cross-view instance segmentation by integrating a geometry-aware VGGT encoder with a novel Union Segmentation Head to overcome pixel-level projection drift. It achieved state-of-the-art average IoU scores of 67.7% for Ego→Exo and 68.0% for Exo→Ego on the Ego-Exo4D benchmark, demonstrating robust generalization without explicit paired cross-view annotations.

Key insights

• Translating high-level geometric consistency from models like VGGT into precise pixel-accurate segmentation requires a specialized segmentation head to correct for projection drift.
• Explicitly fusing semantic mask embeddings with cross-view features and using sparse, geometry-aware point prompts can significantly improve object localization and precise boundary prediction.
• Self-supervised training with diverse single-image augmentations enables a cross-view mask head to learn robust object correspondence without requiring expensive paired ego-exo annotations.

Summary by alphaXiv.