Reproduce “You2Me: Inferring Body Pose in Egocentric Video via First and Second Person Interactions” on OpenResearch

About this paper

Evonne Ng, Donglai Xiang, Hanbyul Joo, Kristen Grauman

Researchers developed You2Me, a method to infer the camera wearer's full 3D body pose from monocular egocentric video by explicitly modeling the temporal action-reaction dynamics between the camera wearer and a visible interactee. This approach achieved lower average joint errors (e.g., 14.3 cm on the Kinect dataset) compared to prior egocentric pose estimation techniques.

Key insights

• The dynamics of human-human interactions, captured through the visible second person's pose, provide crucial information for inferring the largely unobserved first person's 3D body pose, particularly for subtle gestures.
• Formulating pose estimation as a classification problem over a quantized pose space is more robust and accurate than direct regression for egocentric pose inference.
• Explicitly using 2D pose estimates of the visible interactee (second person) significantly improves the accuracy of the camera wearer's pose estimation, outperforming methods relying solely on self-motion or static scene cues.

Summary by alphaXiv.