Reproduce “ZeroPS: High-quality Cross-modal Knowledge Transfer for Zero-Shot 3D Part Segmentation” on OpenResearch

About this paper

Yuheng Xue, Nenglun Chen, Jun Liu, Wenyun Sun

A training-free pipeline named ZeroPS enables high-quality zero-shot 3D part segmentation by transferring knowledge from 2D foundation models like SAM and GLIP via multi-view correspondence. This method achieved 56.0% Average IoU for unlabeled segmentation on PartNetE and 28.5% mAP50 for instance segmentation, significantly improving upon previous zero-shot techniques and narrowing the gap to supervised approaches.

Key insights

• High-quality zero-shot 3D part segmentation can be achieved by effectively leveraging the inherent relationship between multi-view correspondence and the prompt mechanisms of powerful 2D foundation models.
• Iteratively extending 2D segmentation masks generated by a 2D foundation model across multiple viewpoints provides a robust way to construct complete and consistent 3D parts without requiring 3D annotations.
• Combining geometric consistency checks in both 2D and 3D spaces with a multi-view voting refinement strategy significantly improves the reliability and accuracy of cross-modal label transfer for instance segmentation.

Summary by alphaXiv.