Reproduce “Grasping Deformable Objects via Reinforcement Learning with Cross-Modal Attention to Visuo-Tactile Inputs” on OpenResearch

About this paper

Yonghyun Lee, Sungeun Hong, Min-gu Kim, Gyeonghwan Kim, Changjoo Nam

Researchers at Sogang University developed a deep reinforcement learning framework that enables a simple robotic gripper to grasp soft deformable objects stably and safely. The framework leverages a novel cross-modal attention module to effectively fuse visual and tactile sensory inputs, demonstrating superior and more stable learning performance compared to baseline multi-modal and visual-only approaches in simulation.

Key insights

• Multi-modal visuo-tactile sensing is crucial for grasping deformable objects, as relying solely on visual information is insufficient for understanding contact dynamics and achieving stable control.
• Cross-modal attention mechanisms effectively integrate global visual and local tactile information, leading to more stable learning and superior performance in DRL agents compared to simpler fusion strategies.
• Using semantic segmentation masks instead of raw RGB images helps the DRL agent focus on object shapes and robot movement, reducing dependency on visual patterns and improving robustness.

Summary by alphaXiv.