Reproduce “Visual-auditory Extrinsic Contact Estimation” on OpenResearch

About this paper

Xili Yi, Jayjun Lee, Nima Fazeli

VA2Contact introduces a multimodal deep learning framework combining visual and active audio sensing for estimating extrinsic contact location and shape between a robot-held object and the environment. Utilizing a novel real-to-sim audio hallucination technique, the system achieves robust zero-shot transfer to real-world scenarios, boosting contact detection F1-score to 0.94 and IOU for geometry to 0.240, and doubling the success rate in a robotic wiping task.

Key insights

• Active acoustic sensing provides unambiguous evidence of physical contact, effectively resolving visual ambiguities in near-contact situations and compensating for occluded visual information.
• While active audio is highly effective for detecting the *presence* of contact, its combination with optical flow from visual data is crucial for accurately estimating the *location, shape, and size* of the contact patch.
• The proposed real-to-sim audio hallucination technique offers a scalable and effective strategy to bridge the data generation gap for multimodal learning, enabling robust sim-to-real transfer without requiring high-fidelity audio simulation.

Summary by alphaXiv.