Reproduce “Learning Video-Conditioned Policies for Unseen Manipulation Tasks” on OpenResearch

About this paper

Elliot Chane-Sane, Cordelia Schmid, Ivan Laptev

A new method, Video-conditioned Policy learning (ViP), enables robots to perform previously unseen manipulation tasks by interpreting "in-the-wild" human video demonstrations. This approach learns a policy conditioned on video embeddings and employs k-nearest neighbors regression to bridge the human-robot domain gap, achieving an average 71.3% success rate on TableTop tasks without requiring paired human-robot demonstration data during training.

Key insights

• Video embeddings, derived from pre-trained human action recognition models, can effectively serve as task-defining contexts for multi-task robot manipulation policies across the human-robot domain gap.
• Bridging the human-robot domain gap at inference time via k-NN regression in a robot embedding space is crucial for zero-shot generalization, as direct use of human video embeddings for policy conditioning performs poorly.
• A task similarity function learned solely from diverse 'in-the-wild' human videos can successfully compare human and robot videos, enabling zero-shot execution of unseen tasks without requiring paired human-robot demonstration data.

Summary by alphaXiv.