Reproduce “Interaction Replica: Tracking Human-Object Interaction and Scene Changes From Human Motion” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Vladimir Guzov, Julian Chibane, Riccardo Marin, Yannan He, Yunus Saracoglu, Torsten Sattler, Gerard Pons-Moll

The iReplica framework captures comprehensive human-object interactions and tracks dynamic scene changes using only wearable body-mounted IMU sensors and a head-mounted camera. It generates visually plausible digital replicas of interactions, achieving an average object localization error of 6.88 cm and angular error of 7.13 degrees at interaction end, and significantly improving human body localization compared to previous egocentric tracking methods.

Key insights

  • Human-scene contacts possess distinctive characteristics detectable from human pose sequences alone, providing a reliable signal for interaction even when direct visual cues are absent from egocentric cameras.
  • Precisely detected contact timestamps and associated points can be used to regularize and significantly improve both human localization and the estimation of object motion, ensuring visual plausibility during interaction.
  • Modeling object motion based on pre-defined degrees of freedom (e.g., hinged, sliding) in a known environment, coupled with inferred human contact, enables accurate object state updates without external object trackers.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.