Reproduce “Learning Generalizable Robot Policy with Human Demonstration Video as a Prompt” on OpenResearch

About this paper

Xiang Zhu, Yichen Liu, Hezhong Li, Jianyu Chen

A two-stage framework enables robots to learn dexterous manipulation skills by interpreting human demonstration videos as prompts for new tasks. This method leverages a cross-prediction video generation model to learn generalizable task representations and a ProtoDiffusion Contrastive Policy, demonstrating improved generalization to novel tasks, objects, and skills on real-world robot platforms.

Key insights

• The cross-prediction mechanism effectively transfers task and skill knowledge between human and robot embodiments, allowing generated videos to depict tasks by different agents.
• Learned representations from human and robot videos exhibit vector arithmetic properties, enabling manipulation and composition of skills similar to word embeddings.
• Integrating a prototypical contrastive loss within the diffusion policy significantly improves skill separation and multi-skill imitation performance by fostering robust, task-aware feature clustering.

Summary by alphaXiv.