Reproduce “ConViTac: Aligning Visual-Tactile Fusion with Contrastive Representations” on OpenResearch

About this paper

Zhiyuan Wu, Yongqiang Zhao, Shan Luo

King's College London researchers developed ConViTac, a visual-tactile fusion network that uses Contrastive Embedding Conditioning to align visual and tactile features by integrating self-supervised contrastive representations into a supervised framework. This approach led to up to 12.0% higher accuracy in material classification and a new state-of-the-art 84.3% accuracy in grasping success prediction, enhancing robotic perception for complex tasks.

Key insights

• Human multisensory integration, where touch refines visual understanding of specific areas, motivates the need for active alignment in robotic perception.
• Contrastive learning excels at learning robust joint representations and establishing semantic correspondence between visual and tactile modalities.
• Directly conditioning supervised feature fusion with frozen self-supervised contrastive embeddings effectively aligns cross-modal features, bridging the gap between representation learning and task-specific optimization.

Summary by alphaXiv.