Reproduce “TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understanding” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yun Liu, Haolin Yang, Xu Si, Ling Liu, Zipeng Li, Yuxiang Zhang, Yebin Liu, Li Yi

The TACO dataset, featuring 2.5K bimanual interaction sequences and 5.2M frames, enables benchmarking generalizable hand-object understanding across novel tool geometries, interaction triplets, and tool categories. Its automatic acquisition pipeline provides detailed 4D annotations for 131 unique tool-action-object compositions, covering 20 object categories and 15 daily actions.

Key insights

  • Existing methods for action recognition, motion forecasting, and grasp synthesis exhibit a significant performance drop when generalizing to novel interaction triplets or tool categories compared to novel object geometries.
  • Modeling the interaction context (e.g., the second hand and target object) is crucial for cooperative grasp synthesis, significantly improving contact and reducing collisions.
  • Generalization of motions for dominant interaction entities (like the tool-wielding hand and the tool itself) proves more challenging than for non-dominant entities (left hand, target object).

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.