Reproduce “ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Data” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zeyi Liu, Cheng Chi, Eric Cousineau, Naveen Kuppuswamy, Benjamin Burchfiel, Shuran Song
ManiWAV introduces a system that learns robot manipulation skills from in-the-wild audio-visual human demonstrations, leveraging contact microphones and a robust policy learning framework. This approach enables robots to perform contact-rich tasks with significantly higher success rates, such as 85% for wiping and 90% for pouring, compared to vision-only methods.
Key insights
- Audio signals offer rich and often unique information about contact dynamics, object properties, and internal states (e.g., cup fullness), which are critical for precise robot manipulation and are frequently ambiguous or absent in visual data.
- Collecting diverse, "in-the-wild" human demonstrations using a low-cost, portable "ear-in-hand" device dramatically improves policy generalization compared to lab-only data.
- Effective audio data augmentation, which simulates real-robot noise during training, is essential for building robust policies that can reliably leverage audio information in noisy deployment environments.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.