Reproduce “Robot Learning with Sensorimotor Pre-training” on OpenResearch

About this paper

Ilija Radosavovic, Baifeng Shi, Letian Fu, Ken Goldberg, Trevor Darrell, Jitendra Malik

Researchers at the University of California, Berkeley developed Robot Pre-Training (RPT), a self-supervised sensorimotor pre-training model that learns general and transferable representations from real-world multimodal robot interaction data through a masked prediction task. RPT consistently outperforms training from scratch with up to a 2x increase in success rate on manipulation tasks, scales effectively, and enables robust transfer across tasks, environments, and different robot platforms.

Key insights

• Learning to predict masked sensorimotor content across modalities in robot trajectories can build robust models of the physical world, forming a basis for intelligent robot action.
• Operating on latent visual representations derived from pre-trained vision encoders makes the prediction task tractable and allows for scalable, real-time inference on physical robots.
• Generalizable sensorimotor representations can be learned directly from raw, unlabeled real-world robot interactions, reducing reliance on extensive task-specific or human-labeled data.

Summary by alphaXiv.