Reproduce “MimicFunc: Imitating Tool Manipulation from a Single Human Video via Functional Correspondence” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Chao Tang, Anxing Xiao, Yuhong Deng, Tianrun Hu, Wenlong Dong, Hanbo Zhang, David Hsu, Hong Zhang

A framework named MimicFunc enables robots to imitate complex tool manipulation skills from a single human video and generalize them to novel tools with varying geometries but equivalent functions. This approach, leveraging a 'function frame' representation, leads to an average success rate of 79.5% for novel tool generalization and improves visuomotor policy training by generating high-quality demonstration data.

Key insights

  • Focusing on functional correspondence, rather than purely geometric or visual similarities, allows robots to robustly generalize tool manipulation to novel tools with diverse appearances.
  • A "function frame," a function-centric local coordinate system defined by functional keypoints, effectively abstracts tool functionality, enabling consistent skill transfer across varied tools.
  • The use of Vision-Language Models (VLMs) for coarse region proposals, combined with dense semantic correspondence, is effective for accurate functional keypoint transfer to new objects.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.