Reproduce “I'M HOI: Inertia-aware Monocular Capture of 3D Human-Object Interactions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chengfeng Zhao, Juze Zhang, Jiashen Du, Ziwei Shan, Junye Wang, Jingyi Yu, Jingya Wang, Lan Xu
A new system, 'I’m-HOI', developed by researchers at ShanghaiTech University, enables robust 3D human-object interaction capture using a single RGB camera and object-mounted IMUs. It achieves superior accuracy, outperforming baselines by approximately 15 cm in challenging dynamic scenarios, and processes frames at a significantly faster rate of 0.5 seconds.
Key insights
- Combining monocular RGB with object-mounted IMUs effectively addresses the limitations of vision-only methods, providing robustness against occlusions and depth ambiguities in dynamic human-object interactions (HOI).
- A multi-stage deep learning pipeline, integrating holistic tracking with a diffusion-based refinement, can accurately fuse multi-modal data and generate realistic human-object interactions, including fine-grained hand motions.
- The explicit incorporation of IMU data, particularly acceleration constraints, is crucial for improving object tracking precision and ensuring physical plausibility, especially when objects are occluded.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.