Reproduce “PACT: End-to-End Learning of Human Pose, Contacts, and Forces from Video” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Rikhat Akizhanov, Yangsong Zhang, Nikolai Kaliazin, Peter Wolf, Yoshihiko Nakamura, Pascal Fua, Fabio Pizzati, Ivan Laptev

PACT estimates pose, scene contacts, and forces jointly from video. It builds on a visual reconstruction model, adds contact-force tokens, and combines image cues with world-space motion in a temporal transformer, using physics-based training to encourage consistency. On FORCEWALL, it reports 85.5 N force error and 0.725 magnitude correlation, for rigid, static scenes and excluding object and muscle forces.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.