Reproduce “Affordances from Human Videos as a Versatile Representation for Robotics” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, Deepak Pathak

A new method, Vision-Robotics Bridge (VRB), enables robots to learn object interaction skills by extracting actionable affordance representations (contact points and post-contact trajectories) directly from large-scale human videos. The approach successfully integrates with and enhances various robot learning paradigms, demonstrating improved performance across diverse real-world tasks and robotic platforms.

Key insights

  • Robot-agnostic contact points and post-contact trajectories derived from human videos provide a versatile, actionable representation for robotic manipulation.
  • Large-scale passive human video datasets can serve as a rich and scalable source for learning generalizable interaction skills, circumventing the need for expensive robot-specific data.
  • The learned affordance representation can effectively bootstrap and enhance multiple robot learning paradigms, including offline imitation, reward-free exploration, goal-conditioned learning, and defining structured action spaces.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.