Reproduce “Hybrid 3D Human Pose Estimation with Monocular Video and Sparse IMUs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yiming Bao, Xu Zhao, Dahong Qian
A new hybrid framework, RTOF, improves 3D human pose estimation by integrating monocular video with sparse IMUs, yielding accurate and temporally smooth results in real-time. This method achieved a Mean Per Joint Position Error (MPJPE) of 33.7mm on the Total Capture dataset, demonstrating robust performance even surpassing some multi-view methods with ground truth 2D inputs (23.2mm MPJPE).
Key insights
- Direct integration of raw IMU orientations and accelerations into a kinematic human model in an Inertial-Guided Inverse Kinematic layer effectively corrects local joint accuracy.
- A fragment-based temporal optimization, driven by a hybrid energy function that incorporates both visual and raw inertial data, is crucial for eliminating temporal jitters and producing biomechanically plausible motion.
- Combining monocular video with sparse IMUs can yield superior 3D human pose estimation results, outperforming methods reliant on more extensive multi-view camera setups.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.