Reproduce “ViTacGen: Robotic Pushing with Vision-to-Touch Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhiyuan Wu, Yijiong Lin, Yongqiang Zhao, Xuyang Zhang, Zhuo Chen, Nathan Lepora, Shan Luo
ViTacGen enables robotic pushing without tactile sensors at deployment by predicting contact-depth maps from camera input. It learns from paired visual and tactile data in simulation, then combines generated maps with visual information in a pushing policy. In 50 real-world mug trials, it achieved 86% success and 1.8 ± 0.7 cm mean final distance error, versus 10% and 7.2 ± 2.1 cm for a visual-only baseline; success required ending within 2.5 cm of the goal.
Key insights
- High-quality tactile information can be accurately synthesized from visual data, enabling tactile-informed policies without physical tactile sensors.
- Effective multi-modal feature fusion through attention mechanisms and cross-modal alignment via contrastive learning significantly enhances robotic manipulation policy performance.
- Utilizing standardized tactile representations and extensive domain randomization in simulation facilitates robust zero-shot transfer to real-world visual-only systems.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.