Reproduce “Interacting Hand-Object Pose Estimation via Dense Mutual Attention” on OpenResearch

About this paper

Rong Wang, Wei Mao, Hongdong Li

Researchers at The Australian National University developed a method for 3D hand-object pose estimation using a dense mutual attention mechanism within a Graph Convolutional Network. The approach achieved state-of-the-art accuracy on HO3D v2 and DexYCB datasets, reducing hand MJE to 2.38 cm on HO3D and object MCE to 3.26 cm on DexYCB, while maintaining a real-time inference speed of 34 FPS.

Key insights

• Modeling dense, vertex-level mutual attention between hand and object meshes is crucial for capturing fine-grained physical interactions and significantly improving pose plausibility.
• Graph Convolutional Networks effectively refine initial rough hand and object poses by learning intra-graph dependencies and integrating features from the mutual attention layer.
• Attention map visualizations confirm that the mutual attention mechanism effectively identifies and leverages contact priors, with higher attention values observed in actual contact regions between the hand and object.

Summary by alphaXiv.