Reproduce “Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks” on OpenResearch

About this paper

Michelle A. Lee, Yuke Zhu, Peter Zachares, Matthew Tan, Krishnan Srinivasan, Silvio Savarese, Li Fei-Fei, Animesh Garg and 1 more

Researchers at Stanford University developed a two-stage learning framework that combines visual and haptic feedback to enable robots to perform contact-rich manipulation tasks with high sample efficiency and generalization. Their method achieved up to 98% success in peg insertion tasks, even for novel geometries, by learning a compact multimodal representation using self-supervised objectives.

Key insights

• Combining vision and touch feedback is crucial and complementary for high-precision, contact-rich manipulation tasks.
• Self-supervised learning of a compact, control-relevant multimodal representation significantly improves the sample efficiency of subsequent reinforcement learning.
• The 'pairing prediction' objective is particularly effective for learning meaningful correlations and temporal alignment between heterogeneous sensory modalities.
• Variational inference provides beneficial regularization for multimodal representation learning, preventing overfitting.

Summary by alphaXiv.