Reproduce “Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Junuk Cha, Jihyeon Kim, Jae Shin Yoon, Seungryul Baek

This paper introduces Text2HOI, a framework that generates realistic 3D hand-object interaction motions from a text prompt and an object mesh, eliminating the need for object trajectories or initial hand poses. The method surpasses existing baselines in physical realism, achieving a score of 0.8839 on the GRAB dataset compared to 0.7382 for the next best method, and demonstrates strong generalization to unseen objects.

Key insights

  • Decomposing the complex hand-object interaction problem into contact map prediction and motion generation, followed by refinement, addresses data scarcity and improves realism.
  • Integrating geometric losses (distance map, relative orientation) and a specialized hand refinement network with penetration and contact losses significantly enhances physical plausibility of interactions.
  • The use of novel frame-wise and agent-wise positional encodings in the Transformer, combined with conditional inputs like predicted contact maps and object scale, are critical for generating accurate and diverse interactions.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.