Reproduce “Sparsh: Self-supervised touch representations for vision-based tactile sensing” on OpenResearch

About this paper

Carolina Higuera, Akash Sharma, Chaithanya Krishna Bodduluri, Taosha Fan, Patrick Lancaster, Mrinal Kalakrishnan, Michael Kaess, Byron Boots and 3 more

Sparsh introduces a family of self-supervised touch representations for vision-based tactile sensors, trained on over 460,000 unlabeled tactile images to reduce reliance on costly labeled data. The work also establishes TacBench, a standardized benchmark for evaluating these representations, demonstrating Sparsh's consistent outperformance of end-to-end models by an average of 95.1% across diverse manipulation tasks.

Key insights

• Self-supervised learning on large datasets of unlabeled tactile images produces robust, general-purpose representations that generalize across tasks and sensor types.
• Learning tactile representations in latent space (e.g., via DINO or JEPA) is more effective than pixel-level reconstruction (MAE), improving robustness to noise and variations in lighting or sensor markings.
• Pre-trained tactile representations significantly reduce the need for large, labeled datasets in downstream tasks, enabling efficient adaptation to new manipulation challenges and facilitating cross-sensor transfer.

Summary by alphaXiv.