Reproduce “Human-to-Robot Imitation in the Wild” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shikhar Bahl, Abhinav Gupta, Deepak Pathak
A robot learning algorithm enables robots to acquire complex manipulation skills by observing unstructured human videos, demonstrating successful application across 20 diverse real-world tasks with efficient iterative refinement. The system effectively addresses the human-robot embodiment mismatch by learning from environmental changes rather than direct action imitation.
Key insights
- Human videos, despite embodiment mismatch, can provide valuable initial priors for robot skill learning in unstructured environments.
- An agent-agnostic video alignment objective, utilizing inpainting and action recognition embeddings, effectively bridges the human-robot embodiment gap by focusing on environmental changes rather than specific movements.
- An iterative "Watch, Repeat, Improve" paradigm combining residual learning, conditional variational auto-encoders (CVAEs), and a sampling-based optimization enables safe and sample-efficient real-world policy refinement.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.