Reproduce “Self-Supervised Visuo-Tactile Pretraining to Locate and Follow Garment Features” on OpenResearch

About this paper

Justin Kerr, Huang Huang, Albert Wilcox, Ryan Hoque, Jeffrey Ichnowski, Roberto Calandra, Ken Goldberg

Researchers developed Self-Supervised Visuo-Tactile Pretraining (SSVTP), a framework that autonomously collects spatially aligned vision and tactile data to learn generalizable representations for robotic manipulation of deformable garment features. This framework enables robots to perform diverse perception and control tasks, such as anomaly detection and edge following, in a zero-shot manner, outperforming baselines and reducing reliance on human-labeled data.

Key insights

• Self-supervised pretraining with precisely spatially aligned real-world visuo-tactile data can effectively learn flexible and generalizable representations for deformable object manipulation.
• Decoupling the representation of texture (through a rotation-invariant latent space) from the prediction of orientation (via a dedicated network) is crucial for robust task performance, especially in tasks like edge following.
• The learned visuo-tactile representations enable zero-shot execution of a wide range of perception and control tasks, reducing the need for task-specific fine-tuning or additional labeled data.

Summary by alphaXiv.