Reproduce “Giving Robots a Hand: Learning Generalizable Manipulation with Eye-in-Hand Human Video Demonstrations” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Moo Jin Kim, Jiajun Wu, Chelsea Finn

Researchers at Stanford University developed a framework to train generalizable robot manipulation policies by integrating eye-in-hand human video demonstrations. This approach leverages image masking and grasp state conditioning to bridge the visual domain gap, resulting in an average absolute success rate improvement of 58% across unseen environments and longer-horizon tasks compared to robot-only training.

Key insights

  • A fixed image masking scheme in conjunction with eye-in-hand cameras can effectively reduce the visual domain gap between human and robot data without requiring complex explicit domain adaptation methods.
  • Explicitly conditioning the policy on the grasp state is crucial for accurate manipulation, as it provides essential proprioceptive information that is otherwise obscured by image masking.
  • Combining a small set of robot demonstrations with large volumes of inferred-action human video demonstrations significantly improves both environment and task generalization for robot manipulation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.