Reproduce “Cross-Modal Visuo-Tactile Object Perception” on OpenResearch

About this paper

Anirvan Dutta, Simone Tasciotti, Claudia Cusseddu, Ang Li, Panayiota Poirazi, Julijana Gjorgjieva, Etienne Burdet, Patrick van der Smagt and 1 more

The Cross-Modal Latent Filter (CMLF) is a hierarchical deep state-space model that enables unsupervised inference of latent physical properties for novel, non-rigid objects using combined visual and tactile data on a robotic platform. This framework achieved faster and more accurate property estimation and improved robustness to sensory noise compared to baseline approaches, while demonstrating behaviors consistent with biological multi-sensory perception.

Key insights

• Explicitly modeling the temporal evolution of beliefs and supporting bidirectional information transfer between modalities, inspired by human active inference, leads to more robust and efficient property estimation.
• Factorizing the latent space into directly observable (e.g., pose) and indirectly observable (e.g., stiffness) components and coupling them via a shared dynamics model enhances interpretability and inference accuracy.
• Delaying the activation of cross-modal priors until unimodal latent representations stabilize improves learning effectiveness, mirroring biological developmental processes in multi-sensory integration.

Summary by alphaXiv.