Reproduce “Semantically Grounded Object Matching for Robust Robotic Scene Rearrangement” on OpenResearch

About this paper

Walter Goodwin, Sagar Vaze, Ioannis Havoutis, Ingmar Posner

Goodwin et al. introduce a semantically grounded object matching method, CLIP-SemFeat, for robust robotic scene rearrangement that leverages vision-language models to match objects across different instances of the same semantic class. This approach significantly outperforms purely visual methods in cross-instance matching tasks, achieving up to 77.4% accuracy in real robot experiments for rearrangement.

Key insights

• Purely visual object matching methods, including those based on deep features from ImageNet-trained CNNs or CLIP's visual encoder, experience substantial performance degradation in cross-instance scenarios and with increasing visual shifts.
• Leveraging semantic understanding from vision-language models (like CLIP) effectively enables robust object matching across different instances of the same semantic object class, overcoming the
• same-instance
• limitation.
• The formulation of textual prompts (prompt engineering) is critical for maximizing CLIP's zero-shot classification performance and overall matching accuracy in visual-semantic grounding tasks.

Summary by alphaXiv.