Reproduce “Structured World Models from Human Videos” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Russell Mendonca, Shikhar Bahl, Deepak Pathak
Researchers from Carnegie Mellon University introduce Structured World Models for Intentionality (SWIM), a framework that enables robots to learn complex manipulation skills with high sample efficiency by pre-training a world model on large-scale human videos using a morphology-invariant visual affordance action space. The system achieves an average of ~79% success across diverse real-world tasks using less than 30 minutes of robot interaction data for fine-tuning.
Key insights
- Human videos, despite the morphology gap, can provide actionable dynamics for robots by translating human interactions into structured visual affordances.
- Pre-training world models on internet-scale human videos drastically improves sample efficiency and enables rapid adaptation to new robot embodiments and environments.
- A jointly trained world model across diverse tasks and robots benefits from shared dynamics and physics learned from human interactions, leading to better generalization than single-task models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.