Reproduce “In-N-On: Scaling Egocentric Manipulation with in-the-wild and on-task Data” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xiongyi Cai, Ri-Zhao Qiu, Geng Chen, Lai Wei, Isabella Liu, Tianshu Huang, Xuxin Cheng, Xiaolong Wang

UC San Diego researchers developed a scalable method called "In-N-On" that leverages large-scale egocentric human data, categorized into "in-the-wild" and "on-task" types, to train foundational robot manipulation policies. This approach enables emergent capabilities such as zero-shot language following for instructions unseen by the robot and 1-shot learning for complex bimanual tasks, showing improved performance and robustness on humanoid robots.

Key insights

  • A systematic, two-stage approach leveraging both diverse "in-the-wild" and task-specific "on-task" human data is essential for emergent robot capabilities like zero-shot language following.
  • Explicit domain adaptation through techniques like Gradient Reversal Layer prevents models from "cheating" by merely distinguishing embodiments, ensuring the learning of truly transferable skills.
  • Pre-training with large human datasets provides strong priors that significantly enhance few-shot learning, allowing robots to quickly adapt to new tasks with minimal robot-specific demonstrations.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.