Reproduce “Deep Two-Stream Video Inference for Human Body Pose and Shape Estimation” on OpenResearch

About this paper

Ziwen Li, Bo Xu, Han Huang, Cheng Lu, Yandong Guo

The DTS-VIBE framework from OPPO Research Institute and Xmotors introduces a deep two-stream architecture for 3D human pose and shape estimation from monocular video, fusing RGB and virtually estimated optical flow features with Transformer-based temporal modeling and a novel flow loss. This approach achieves state-of-the-art accuracy on 3DPW, Human3.6M, and MPI-INF-3DHP datasets, while significantly reducing the acceleration error on Human3.6M by 67.3%.

Key insights

• Integrating 'virtual' optical flow, derived from RGB video, effectively enhances motion understanding and improves 3D pose and shape estimation accuracy, offering an economical multi-modal solution.
• Transformer architectures significantly improve temporal consistency and accuracy in 3D human reconstruction by capturing long-range temporal correlations, outperforming traditional GRU-based networks.
• Explicitly enforcing consistency between predicted 2D joint movements and optical flow through a dedicated 'flow loss' further refines pose predictions and enhances temporal smoothness.

Summary by alphaXiv.