Reproduce “Let Robots Feel Your Touch: Visuo-Tactile Cortical Alignment for Embodied Mirror Resonance” on OpenResearch

About this paper

Tianfang Zhu, Ning An, Rui Wang, Jiasi Gao, Qingming Luo, Anan Li, Guyue Zhou

Researchers developed a computational framework that translates neuroscientific principles of cortical alignment to enable robots to predict tactile sensations from visual observations. This allowed a robotic hand to generate self-referenced tactile predictions and physical responses when observing a human hand being touched, demonstrating cross-domain generalization without requiring tactile ground truth for the human domain.

Key insights

• Explicitly enforcing structural alignment between visual and tactile modalities through multi-level constraints is essential for learning robust and physically plausible cross-modal mappings in robots, moving beyond reconstruction losses.
• The alignment mechanism regularizes and simplifies the visual-to-tactile mapping, guiding the model to focus on contact-relevant physical structures and generating coherent latent representations.
• Cross-domain generalization for tactile prediction can be achieved by aligning the visual features of an unseen domain (e.g., human hands) to a pre-established robotic visuo-tactile manifold, enabling inference without direct tactile supervision in the new domain.

Summary by alphaXiv.