Reproduce “POMA-3D: The Point Map Way to 3D Scene Understanding” on OpenResearch

About this paper

Ye Mao, Weixun Luo, Ranran Huang, Junpeng Jing, Krystian Mikolajczyk

POMA-3D introduces a self-supervised model learning 3D scene representations using "point maps"—3D coordinates encoded on a 2D grid—to bridge the gap between 2D foundation models and 3D data. The approach achieves improved perplexity on 3D Question Answering benchmarks, outperforming SceneVerse by 6.2% on SQA3D, alongside strong performance in scene retrieval, embodied navigation, and coarse localization tasks using only geometric inputs.

Key insights

• Point maps effectively bridge the modality gap, enabling a more efficient transfer of rich semantic priors from 2D foundation models into the 3D domain for robust 3D scene understanding.
• Leveraging pseudo point maps derived from large-scale 2D images during a warm-up pretraining stage is crucial for scalability and substantially enhances the learning of foundational 3D features.
• The POMA-JEPA objective, particularly with Chamfer Distance, successfully enforces geometric consistency across multiple views, yielding 3D representations that are robust and invariant to viewpoint changes.

Summary by alphaXiv.