Reproduce “Generalist Robot Manipulation beyond Action Labeled Data” on OpenResearch

About this paper

Alexander Spiridonov, Jan-Nico Zaech, Nikolay Nikolov, Luc Van Gool, Danda Pani Paudel

MotoVLA introduces a generalist robot manipulation policy that learns embodiment-agnostic motion priors from extensive unlabeled human and robot videos using dynamic 3D point clouds. This policy then aligns these priors with executable robot actions, demonstrating up to 68.2% success in simulator tasks and facilitating direct skill transfer from human demonstrations to real-world robot operations.

Key insights

• Dynamic 3D point clouds provide an effective embodiment-agnostic representation for transferring manipulation skills from diverse, unlabeled videos across human and robot domains.
• Pre-training with unlabeled human demonstrations directly enables zero-shot skill transfer to robots for tasks present in the human video dataset, demonstrating effective cross-embodiment learning.
• The two-stage training framework, separating self-supervised motion learning from supervised action alignment, is crucial for leveraging unlabeled data efficiently and achieving out-of-action domain generalization.

Summary by alphaXiv.