Reproduce “End-to-End HOI Reconstruction Transformer with Graph-based Encoding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhenrong Wang, Qi Zheng, Sihan Ma, Maosheng Ye, Yibing Zhan, Dongjiang Li
Researchers from Shenzhen University and collaborators introduced HOI-TG, an end-to-end transformer framework enhanced with graph-based encoding to implicitly reconstruct 3D human and object meshes and their interactions from a single monocular image. The model achieved state-of-the-art results on BEHAVE and InterCap datasets, improving human Chamfer Distance by 8.0% and object Chamfer Distance by 5.0% on BEHAVE compared to prior methods, while also demonstrating faster inference at 0.208 seconds.
Key insights
- Implicitly learning human-object interactions within an end-to-end transformer framework can effectively resolve the inherent conflict between global mesh reconstruction and fine-grained local interaction details.
- The integration of graph convolutional structures with the transformer's self-attention mechanism is crucial for balancing global interaction modeling with the preservation of local topological relationships and distinct entity boundaries.
- Providing discriminative 3D query features, derived from initial mesh estimations and image features, significantly enhances the transformer's ability to reconstruct complex interaction poses and differentiate between 3D modalities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.