Reproduce “Visuo-Tactile Transformers for Manipulation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yizhou Chen, Andrea Sipos, Mark Van der Merwe, Nima Fazeli
Researchers at the University of Michigan introduce Visuo-Tactile Transformers (VTTs), an architecture for multimodal representation learning that fuses visual and tactile feedback using self- and cross-modal attention. This approach enables robots to learn more sample-efficiently, achieving higher success rates and improved numerical stability in contact-rich manipulation tasks compared to existing baselines.
Key insights
- The explicit use of cross-modal attention in VTT allows tactile information to spatially guide visual processing, creating a more effective and localized understanding of robot-environment interactions.
- Auxiliary tasks like predicting contact state and temporal alignment are critical for the model's sample efficiency and robustness, preventing miscorrelation between visual and tactile inputs.
- VTT's learned attention maps dynamically shift focus from general object/end-effector tracking in contact-free phases to specific contact regions upon interaction, with a concurrent increase in tactile attention.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.