Reproduce “ViTaS: Visual Tactile Soft Fusion Contrastive Learning for Visuomotor Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yufeng Tian, Shuiqi Cheng, Tianming Wei, Tianxing Zhou, Yuanhang Zhang, Zixian Liu, Qianwei Han, Zhecheng Yuan and 1 more
ViTaS introduces a visuo-tactile representation learning framework that leverages Soft Fusion Contrastive Learning and Conditional VAE integration to improve visuomotor control. The method achieves state-of-the-art success rates across 9 simulated RL tasks, outperforms baselines in imitation learning, and demonstrates superior generalization and robustness in real-world robot manipulation.
Key insights
- Effective visuo-tactile fusion requires explicitly addressing both feature alignment (structural similarity) and complementarity (distinctive information) between modalities.
- Soft Fusion Contrastive Learning, which mines top-K similar samples across modalities, provides a more robust and effective feature alignment than conventional instance- or time-based contrastive methods.
- Leveraging CVAEs to reconstruct visual input conditioned on fused visuo-tactile features enables the system to compensate for visual ambiguities and occlusions by drawing upon tactile context.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.