Reproduce “PTLD: Sim-to-real Privileged Tactile Latent Distillation for Dexterous Manipulation” on OpenResearch

About this paper

Rosy Chen, Mustafa Mukadam, Michael Kaess, Tingfan Wu, Francois R Hogan, Jitendra Malik, Akash Sharma

PTLD (Privileged Tactile Latent Distillation) enables learning dexterous manipulation policies with tactile sensing without requiring explicit tactile simulation. It leverages external pose tracking in the real world to distill knowledge from a privileged simulated policy into a tactile-only policy, achieving superior performance on in-hand rotation (182% improvement) and reorientation tasks (57% improvement in goals reached).

Key insights

• Effective tactile policies can be learned without direct tactile simulation by using privileged sensors as an interface for sim-to-real knowledge transfer and distillation.
• Online latent distillation within an AAC framework streamlines the simulation phase into a single, more efficient training step compared to traditional two-stage distillation.
• Tactile feedback provides superior information for object state estimation, particularly in disambiguating dynamic contact sequences and slippage, outperforming proprioception alone.

Summary by alphaXiv.