Reproduce “Multi-HMR 2: Multi-Person Camera-Centric Human Detection, Mesh Recovery and Tracking” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Guénolé Fiche, Philippe Weinzaepfel, Romain Brégier, Fabien Baradel
Multi-HMR 2.a jointly predicts camera parameters and multiple people’s 3D body meshes, using scene-wide queries and 2D-location matching before costly mesh computation. This reduces forward-backward memory from 27 GB to 7 GB. On Harmony4D, pair-aligned joint error is 99.8 mm versus 144.0 mm for fixed-camera SAT-HMR, but EMDB Abs-PVE is worse: 273.6 versus 168.1 mm.
Key insights
- The proposed 2D-based Hungarian matching strategy for training significantly reduces computational cost and memory usage while maintaining accuracy, making the framework more efficient and enabling the use of diverse datasets.
- Directly predicting camera intrinsics (vertical FOV) from an image encoder provides sufficient accuracy for metric 3D localization in varied camera environments, comparable to specialized FOV estimation methods.
- Distilling identity features from foundation models like SAM2 into the HMR framework allows for effective multi-person tracking without explicit video supervision, enabling identity association purely from still image data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.