Reproduce “EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Siwei Zhang, Qianli Ma, Yan Zhang, Zhiyin Qian, Taein Kwon, Marc Pollefeys, Federica Bogo, Siyu Tang

Researchers from ETH Zürich and Microsoft introduce EgoBody, a large-scale, multi-modal egocentric dataset providing high-quality 3D human shape and motion ground truth for two interacting subjects from head-mounted devices. This dataset enables existing models to achieve significantly lower 3D joint error, with fine-tuned state-of-the-art methods showing up to a 42% reduction in MPJPE when processing egocentric views.

Key insights

  • State-of-the-art 3D human pose and shape estimation methods exhibit a substantial performance drop, with an average of 77% higher 3D joint error, when evaluated on egocentric data compared to third-person views.
  • Severe body truncation, particularly of lower body parts, and motion blur are identified as primary factors degrading performance in egocentric perception.
  • The camera wearer's eye gaze is predominantly focused on the interactee's upper body, correlating with higher visibility in egocentric frames and influencing attention patterns.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.