Reproduce “SANPO: A Scene Understanding, Accessibility and Human Navigation Dataset” on OpenResearch

About this paper

Sagar M. Waghmare, Kimberly Wilber, Dave Hawkey, Xuan Yang, Matthew Wilson, Stephanie Debats, Cattalyya Nuengsigkapian, Astuti Sharma and 4 more

Google Research introduced SANPO, a large-scale multimodal dataset combining real-world and synthetic video data for egocentric human navigation, featuring dense panoptic segmentation and depth maps. The dataset addresses the critical lack of appropriate training data for assistive technologies for the visually impaired, with benchmarks revealing a significant domain gap for existing models and demonstrating performance improvements through synthetic-to-real domain adaptation.

Key insights

• State-of-the-art models trained on vehicle-centric datasets experience a substantial performance drop (e.g., Kmax-Deeplab mIoU from 83.5 to 42.4) when applied to SANPO-Real, indicating a large domain gap.
• Machine-propagated panoptic segmentation annotations, validated with a mean J&F score of 0.892, effectively augment human annotations without compromising model performance.
• Integrating synthetic data through pretraining or combined training consistently improves panoptic segmentation results, suggesting its utility for domain adaptation and enhancing model robustness.

Summary by alphaXiv.